Adaptive optimal output tracking of continuous-time systems via output-feedback-based reinforcement learning

Abstract

Reinforcement learning provides a powerful tool for designing a satisfactory controller through interactions with the environment. Although off-policy learning algorithms were recently designed for tracking problems, most of these results either are full-state feedback or have bounded control errors, which may not be flexible or desirable for engineering problems in the real world. To address these problems, we propose an output-feedback-based reinforcement learning approach that allows us to find the optimal control solution using input–output data and ensure the asymptotic tracking control of continuous-time systems. More specifically, we first propose a dynamical controller revised from the standard output regulation theory and use it to formulate an optimal output tracking problem. Then, a state observer is used to re-express the system state. Consequently, we address the rank issue of the parameterization matrix and analyze the state re-expression error that are crucial for transforming the off-policy learning into an output-feedback form. A comprehensive simulation study is given to demonstrate the effectiveness of the proposed approach.

Introduction

Reinforcement learning (RL) (Lewis, Vrabie, and Syrmos, 2012, Sutton and Barto, 1998, Vrabie et al., 2013) or approximate/adaptive dynamic programming (ADP) (Bertsekas, 1995, Lewis, Vrabie, and Vamvoudakis, 2012, Powell, 2007) has been a powerful technique in computational intelligence by feedbacking interactions with the unknown environment for decision making. From a perspective of the control community, RL integrates adaptive control and optimal control with a feature that an adaptive controller solves an optimal control problem (Lewis & Vrabie, 2009), which is different from that of model-based methods documented in Ioannou and Sun (2012) and Krstić et al. (1995).

The RL-based research has been undergoing a period of a major transition from regulation control to tracking control (Chen et al., 2022, Chen and Xie, 2021, Jiang and Jiang, 2017, Lee et al., 2012, Lee et al., 2014, Lewis, Vrabie, and Syrmos, 2012, Vrabie et al., 2013), which covers more practical problems by allowing the system output to track a trajectory. What remains unchanged is that RL algorithms not only stabilize the closed-loop system, but also adaptively achieve a satisfactory tracking performance (Lewis, Vrabie, & Syrmos, 2012). There are two ways to approach the optimal tracking problem via RL for continuous-time (CT) systems, which are the focus of this paper. One way is to build an augmented system containing the original system and command generator, based on which a discounting factor in the cost function and integral RL are used to formulate an optimal tracking control (Modares & Lewis, 2014a). Later, such a result is extended to nonlinear systems (Kamalapurkar et al., 2015, Liu et al., 2017, Luo et al., 2015, Qin et al., 2014), constrained-input systems (Modares & Lewis, 2014b), $H_{\infty }$ control (Modares et al., 2015), and multi-agent systems (Chen et al., 2020, Modares et al., 2018, Modares, Nageshrao, et al., 2016). The other way is to bring the output regulation theory and optimal control together (Krener, 1992, Saberi et al., 2003). Using output regulation theory and ADP, Gao and Jiang (2016) proposed an asymptotic tracking controller, wherein system dynamics are assumed partially known. The work of Chen et al. (2019) removed such prior knowledge of system dynamics and achieved the optimal tracking control using the integral RL and experience replay.

However, most of the RL algorithms mentioned above require full-state measurements. As for control problems in the real world, full states usually are not available. Compared to full-state feedback, output feedback is more flexible and desirable for control implementation.

For output-feedback-based RL, Lewis and Vamvoudakis (2011) parameterized the system state into the Bellman equation and proposed an RL-based method for regulating discrete-time (DT) systems using input and output data. Later, output-feedback-based tracking control (Kiumarsi et al., 2015) was reported for DT systems, based on which $H_{\infty }$ control (Fan et al., 2018) was considered. The work of Rizvi and Lin (2018b) used a method of parameterizing the system state through a DT state observer (Tao, 2003). As for CT systems, Jiang and Jiang (2010) considered an on-policy iteration using a state observer. Using the static output-feedback control and adaptive state observer, a suboptimal controller was given in Zhu et al. (2015). Methods for discrete approximation of CT systems were given in Gao et al. (2016). Based on Modares and Lewis (2014a) and Lewis and Vamvoudakis (2011), Modares, Lewis, and Jiang (2016) considered an output-feedback tracking control for CT systems without a static output-feedback stabilizable condition as required in Zhu et al. (2015), by setting the eigenvalues of the command generator within a predetermined region. Unlike discrete approximation in Gao et al. (2016), Rizvi and Lin (2018a) considered an output-feedback optimal control for regulating CT systems using a state observer to reparameterize system state, which is a CT version of Rizvi and Lin (2018b).

Even though some progress has been made, most of the current output-feedback optimal controls are designed for regulating CT systems. However, in many applications, it is also important to drive the system output to follow a prespecified trajectory with a zero error (see examples in flight control (Stevens et al., 2015) and robotics (Lewis et al., 2003)).

In this paper, we aim to study an optimal output tracking problem of CT systems. We propose an output-feedback-based RL approach, wherein input–output data along system trajectories are used to find the optimal solution without solving output regulation equations and ensure the asymptotic tracking control of CT systems. The contributions of this paper are summarized in the following three aspects.

• Compared to Gao et al. (2016) and Modares, Lewis, and Jiang (2016), we propose an output regulation and off-policy RL-based controller for the optimal output tracking problem.

• Compared to the state re-expression in Rizvi and Lin (2018a), we derive a condition for the parameterization matrix to be full row rank, based on which an approximate optimal control gain is uniquely obtained through output-feedback-based RL.

• The state re-expression error can be made small by running the system for a sufficiently long time before the data collection begins, based on which we propose an additional model-free pre-collection phase to supplement the off-policy learning for CT systems.

The remaining parts of this paper are organized as follows. In Section 2, we formulate an optimal output tracking control problem of CT systems by proposing a dynamical controller and review some preliminaries of optimal control. In Section 3, we present our main result on the optimal output tracking of CT systems via output-feedback-based RL. In Section 4, we provide a simulation example to demonstrate the effectiveness of the proposed approach. Conclusions are drawn in Section 5.

Notations: Throughout this paper, ${ℂ}^{-}$ stands for the open left-half complex plane. For a matrix $X$, $\lambda \left(X\right)$ denotes its spectrum; $X>(\ge )0$ means that the matrix $X$ is positive (semi-positive) definite; and $X<(\le )0$ denotes that $X$ is negative (semi-negative) definite. For a matrix $X\in {\mathbb{R}}^{m\times n}$, $vec\left(X\right)={[x_1^T,x_2^T,\dots ,x_n^T]}^T$ denotes a vector-valued function of a matrix $X$ with $x_i\in {\mathbb{R}}^m$ being the $i$th column of $X$. For a symmetric matrix $X\in {\mathbb{R}}^{m\times m}$, $vecs\left(X\right)=\text{[}{x}_{11},2x_{12},\dots ,2x_{1m},x_{22},2x_{23},\dots ,2x_{m-1,m}$, $x_{m,m}\text{]}{}^T\in {\mathbb{R}}^{\frac{1}{2}m(m+1)}$, where $x_{ij}$ denotes the entry in the $i$th row and $j$th column of the matrix $X$. For a column vector $v\in {\mathbb{R}}^n$, $vecv\left(v\right)={[v_1^2,v_1{v}_2,\dots ,v_1{v}_n,v_2^2,v_2{v}_3,\dots ,v_{n-1}{v}_n,v_n^2]}^T\in {\mathbb{R}}^{\frac{1}{2}n(n+1)}$. $I$ and $O$ are an identity matrix and a zero matrix with appropriate dimensions, respectively. For a matrix $X$, $det\left(X\right)$ denotes the determinant of the matrix and $\text{adj}\left(X\right)$ the adjugate of the matrix. For the matrix pair ($A\in {\mathbb{R}}^{m\times m}$, $X\in {\mathbb{R}}^{m\times n}$), the controllability matrix is defined as ${\mathcal{C}}_X\left(A\right)=[X,AX,A^{2}X,\dots ,A^{m-1}X]$.