This paper proposes a design method of the Q-learning based intelligent supervisory control system (ISCS) for optimal operation of the three step kiln process and a new practical method for improvement of Q-learning convergence. First, the Q-learning based intelligent supervisory control system with two layer-structures is designed to find the on-line optimal set-points of control loops for the kiln process. Next, C4.5 is used to extract automatically the operational experience rules of the human operator from the historical data in the lower layer (i.e., process control layer) and the Q-function value is initialized by using the extracted rules in order to determine the optimal initial point of Q-learning in the higher layer (i.e., supervisory control layer). Hence, the convergence rate of Q-learning is extremely accelerated, so that the hierarchical ISCS can replace the human operator in the kiln process in which trial-and-error operation is not allowed. Through simulations and experiments, Q-learning convergence and the stability of the process operation have been evaluated sufficiently under the variable conditions of the states.

本文提出了一种基于 Q-learning 的智能监控系统 (ISCS) 的设计方法，用于优化三步窑过程的运行，以及一种提高 Q-learning 收敛性的实用新方法。首先，具有两层结构的基于 Q 学习的智能监控系统旨在寻找窑过程控制回路的在线最佳设定点。接下来，利用C4.5从下层（即过程控制层）的历史数据中自动提取操作人员的操作经验规则，并利用提取的规则初始化Q函数值，以确定上层（即监督控制层）中 Q-learning 的最佳初始点。因此，Q-learning 的收敛速度极快，从而分层ISCS可以在不允许试错操作的窑炉过程中代替人工操作。通过仿真和实验，在状态变化的条件下，充分评估了Q-learning收敛性和过程运行的稳定性。