Reinforcement learning based optimal control of batch processes using Monte-Carlo deep deterministic policy gradient with phase segmentation

Highlights

• Design of reward function is suggested for the general economic process control.

• Phase segmentation approach is proposed to address distinct characteristics of various phases of a batch run.

• DDPG algorithm is modified with Monte-Carlo learning for stable agent training.

• Suggested algorithm is applied to a batch polymerization process control problem.

Abstract

Batch process control represents a challenge given its dynamic operation over a large operating envelope. Nonlinear model predictive control (NMPC) is the current standard for optimal control of batch processes. The performance of conventional NMPC can be unsatisfactory in the presence of uncertainties. Reinforcement learning (RL) which can utilize simulation or real operation data is a viable alternative for such problems. To apply RL to batch process control effectively, however, choices such as the reward function design and value update method must be made carefully. This study proposes a phase segmentation approach for the reward function design and value/policy function representation. In addition, the deep deterministic policy gradient algorithm (DDPG) is modified with Monte-Carlo learning to ensure more stable and efficient learning behavior. A case study of a batch polymerization process producing polyols is used to demonstrate the improvement brought by the proposed approach and to highlight further issues.

Introduction

Batch or semi-batch processing is widely used in the process industry, mostly for producing low-volume, high-value added products. Operating condition of a batch process is determined to meet given requirements for end product quality (e.g., composition, particle size and shape) in a manner that assures safety and economic feasibility (e.g., maximization of the productivity or minimization of the cost). However, its inherent features, such as 1) non-stationary operation covering a wide operating envelope 2) consequent exposure of the underlying nonlinear dynamics, and 3) existence of both path and end-point constraints, present a significant challenge to control engineers. These problems are exacerbated by oftentimes significant variabilities in the feedstock quality and condition as well as other process uncertainties (e.g. disturbances, noises, model errors).

For optimal control of batch processes, nonlinear model predictive control (NMPC) has been the most widely studied method (Qin, Badgwell, 2000, Mayne, 2000, Chang, Liu, Henson, 2016). NMPC can be designed in two different forms. First it can be designed to follow a prespecified recipe (e.g., setpoint trajectories), which are determined from an off-line (or run-to-run) optimization. Alternatively, it can be designed to optimize an economic or other performance index directly on-line while respecting process and quality constraints (Rawlings, Amrit, 2009, Ellis, Durand, Christofides, 2014). In this setting, as an economic optimization tends to drive the operating recipe towards active constraints, it is important for the controller to assure constraint satisfaction. While NMPC offers the advantage of including constraints directly in the optimal control calculation, its constraint handling capability can degrade when the model used for the prediction has significant uncertainty (Lucia et al., 2014). Conventional NMPC addresses this problem by reperforming the optimization at each sample time after receiving feedback measurements and also by detuning the controller and tightening the constraints to account for the uncertainty (“back-off”) but this can lead to significant suboptimality (Paulson, Mesbah, 2018, Santos, Bonzanini, Heirung, Mesbah, 2019). Robust NMPC formulations have also appeared which are based on rigorous uncertainty models (e.g., bounded parameter sets, scenario trees) but they come at the cost of significantly increased complexity preventing practical use Morari and H. Lee (1999), Lucia et al. (2013), Thangavel et al. (2018).

In this regard, reinforcement learning (RL), which tries to learn an optimal control policy and value function from data may be worthy of consideration. Learning data can be real operation data or simulation data and such data contain (or can be made to contain) the effect of the uncertainty which will be experienced on-line. Therefore, RL provides a flexible framework wherein a control policy can be learned to address the uncertainty contained in the learning data and its potential has been examined through several set-point tracking control problems (Lee, Lee, 2005, Spielberg, Tulsyan, Lawrence, Loewen, Bhushan Gopaluni, 2019, Kim, Park, Yoo, Oh, Lee, Lee, 2020). For such problems, the set point tracking error was used as the (negative) reward in the training of the RL based controller. On the other hand, just a few studies have examined RL in the context of dynamic optimization of batch process operation (Wilson, Martinez, 1997, Martinez, 1998, Petsagkourakis, Sandoval, Bradford, Zhang, del Rio-Chanona, 2020).

In this paper, we examine the use of RL for on-line dynamic optimization and control of a batch process with high dimensional and continuous state and action spaces. We focus on the issues that arise in such applications, e.g., the choice of reward functions, esp. in handling constraints, construction of time-dependent approximators, and choice of learning algorithm. First of all, we suggest three types of reward functions that combine economic performance index and degree of violation of path and end-point constraints. To address the time dependence of batch operation, a phase segmentation approach is proposed to tailor the reward functions and the function approximators to suit distinct characteristics of the different phases of batch run. For the agent training, deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al., 2016) is adopted as it is known to be effective in handling high dimensional continuous state and action spaces. However, the traditional DDPG algorithm, which uses temporal difference learning, is modified with Monte-Carlo (MC) learning so as to ensure stable learning behavior. This is particularly important for batch process control as the reward function is designed such that violation of the end-point constraints affect the overall reward significantly. The NMPC and different RL control strategies will be examined and compared in a case study that involves a semi-batch polymerization reactor producing polyols (Nie et al., 2013).

In summary, the intended contributions of this paper are in 1) suggesting the RL method for optimal control of batch processes with uncertainty, 2) proposing the phase segmentation approach for designing the reward functions and the function approximators and 3) modifying the DDPG algorithm with MC learning to improve the agent learning in order to better reflect the future value, particularly with respect to the end point constraints satisfactions.

In the next section we briefly introduce MPC, RL, MC vs. TD (Temporal-Difference) learning, and DDPG. Section 3 then proposes a RL based optimal control strategy with specific components like the phase segmentation, and MC-DDPG learning. The case study problem is introduced next and the training environment of the RL based optimal controller is described in Section 4. In Section 5, the improvement by using the phase segmentation approach is demonstrated and the learning results of the conventional DDPG and MC-DDPG algorithms as well as NMPC are compared. In addition, sensitivity analysis with respect to the choice of phase segmentation point and to the hyperparameters used in the reward design are investigated. Section 6 summarizes and concludes the paper.