Active disturbance rejection controller for multi-area interconnected power system based on reinforcement learning

Abstract

In this paper, a method of Active Disturbance Rejection Controller (ADRC) is proposed based on Q-learning of Reinforcement Learning (RL) for multi-area interconnected power system. Excessive changes in load can cause instability to the system. Therefore, the ADRC controller is used to keep the load within rated range for its strong anti-interference performance and Q-learning algorithm to select the adaptive parameters of the controller. Finally, through simulation experiments on traditional and deregulated three-area interconnected power system respectively, the effectiveness of the proposed method is proved and the results show that reinforcement learning can indeed be used to solve the problem of controller parameter adjustment.

Introduction

With the development of the electricity market, people are increasingly demanding electrical energy, which results to the emergence of more and more large-scale power grids and stronger links between power grids areas. However, multi-area interconnected power systems may in turn make the system operate unsafely. This is mainly because the tie-line power between two areas is easily affected by load disturbances, which then can cause overload problems, leading to widespread power outage [1]. Therefore, when the system is subjected to load disturbance, for the safe operation of the system, the system frequency must be kept within a certain range, and the tie-line exchanged power must be stable at planned value. This is the studied problem of load frequency control (LFC) in generalized automatic generation control (AGC) [2]. About LFC, the earliest controller is the traditional PID. The reference [3] applied the PID controller to a two-area interconnected power system, and the simulation results show that the PID controller can effectively control the load frequency. But it can be found that the PID controller is difficult to balance the adjustment time and overshoot. At present, there are two main types of control methods for LFC. One is that optimizing the parameters of PID controller by bionic algorithms such as particle swarm optimization (PSO) [4], [5], fuzzy control algorithms [6], [7], and genetic algorithms (GA) [8], [9], etc. The other is through modern control theory such as predictive control [10], [11], robust control [12], [13], and adaptive control [14], etc. However, if these methods are out of the precise information of the model, the system performance will be greatly affected. Jingqing Han devoted himself to control theory for more than a decade and finally the Active Disturbance Rejection Control (ADRC) emerged [15], [16]. On this basis, Dr. Gao further developed the Linear Active Disturbance Rejection Control (LADRC), and simplified the controller parameters [17], which made the industrialization of the ADRC possible. Since then, the ADRC had been applied and researched by more and more experts and scholars for the characteristic of not requiring the model information of the system. At the same time, ADRC has also demonstrated its superiority in many fields, such as underwater vehicle [18], ship course control [19] and quadrotor control [20]. The tuning and optimization of ADRC controller parameters has always been the key research for scholars and experts, and currently, many Artificial Intelligence algorithms are used to train a set of parameters offline. However, in the actual system, there may be some uncertain factors that lead to changes in the model parameters or structure, which makes the originally controller parameters unable to ensure the good operation of the system. So, it’s necessary to design an adaptable parameter optimization method online. In recent years, the adaptive parameters are usually acquired by the PID controller based on fuzzy control algorithm [21], [22]. However, the formulation of fuzzy rules depends on model information. Therefore, this paper proposes the ADRC controller based on Q-learning which dose not rely on models and can adjust parameters online.

Reinforcement Learning (RL) is a way to find the optimal strategy by the constant interaction between the agent and the environment, where the environment can be uncertain. When the agent ’communicates’ with the environment through the action, the environment returns a current reward to the agent, by which the action can be evaluated [23]. Q-learning is a kind of off-policy algorithm of RL, which was first proposed by Watkins [24] in 1989. For the features of model-free and strong convergence, Q-learning is applied to the robot path planning in complex environments [25], [26] in most cases. In addition, Q-learning has shown success in the problem of optimal tracking control [27], [28]. Nowadays, few studies combine Q-learning and parameter optimization. Reference [7] presents a Fuzzy-Proportional-Integral-Derivative (Fuzzy-PID) controller for a three area interconnected power system, and uses mine blast algorithm (MBA) to optimize the controller parameters, and the method is verified by simulation experiments. This article applies the LADRC controller to the LFC system researched by [7], and uses Q-learning to get the adaptive parameters of LADRC. The simulation results are obtained by the software Matlab, which are compared with the results shown in  [7], and the comparison results of load frequency response imply that Q-learning has good performance in parameter adaptation.

The main novel contributions of this article are summarized as follows:

• A reduced-order LADRC controller is designed to reduce the impact of load frequency disturbances on traditional and deregulated power systems;

• Apply the off-policy algorithm: Q-learning of reinforcement learning to obtain the adaptive parameters of LADRC controller, where the model information can be unknown;

• In order to avoid the unnecessary exploration of actions, the ε of Q-learning algorithm in $\epsilon -greedy$ policy has been improved.