Power allocation is of crucial significance to energy management system in the hybrid construction machinery (HCM). Most of the existing HCM energy management strategies are only formulated based on the predefined rules, which causes the system unable to adapt to the changeable and complicated working conditions, thus seriously limiting the energy saving potential of hybrid technology. In this paper, we build a reinforcement learning-based intelligent energy management architecture for HCM. Given the working conditions and operating characteristics of HCM, a Q-function updating method combining direct learning and indirect learning is proposed to enhance the performance and practicability of reinforcement learning. A virtual world model (VWM) is introduced to approximate the real-world environment and facilitate the identification of data-driven environment, so as to enhance the real-time performance and adaptability of the architecture. Based on the characteristics of HCM working conditions, the load cycle is subdivided, and the stationary Markov chain is employed to yield real-time transfer probability matrices of required power to accelerate the updating of the environment model. An HCM experiment platform is built, in which the typical signal of working condition is sampled for simulation. The results indicate that DYNA-Q based architecture outperforms Q-learning and rule-based strategy (RBS) in terms of adaptivity, real-time performance and optimality. The results also demonstrate that with the proposed architecture, the working condition of internal combustion engine (ICE) and the charge-discharge of ultracapacitor are more rational and efficient.

功率分配对于混合动力工程机械（HCM）的能源管理系统至关重要。现有的大多数HCM能源管理策略仅基于预定义的规则制定，这导致系统无法适应多变和复杂的工作条件，从而严重限制了混合技术的节能潜力。在本文中，我们为HCM构建了一个基于强化学习的智能能源管理架构。针对HCM的工作条件和操作特点，提出了一种将直接学习和间接学习相结合的Q功能更新方法，以提高强化学习的性能和实用性。引入虚拟世界模型（VWM）来逼近真实环境，并有助于识别数据驱动的环境，从而增强体系结构的实时性能和适应性。根据HCM工作条件的特征，可将负载周期进行细分，并使用固定的马尔可夫链产生所需功率的实时传输概率矩阵，以加快环境模型的更新。建立了一个HCM实验平台，在其中采样了典型的工作条件信号进行仿真。结果表明，基于DYNA-Q的体系结构在适应性，实时性能和最优性方面优于Q学习和基于规则的策略（RBS）。结果还表明，对于拟议的架构，