Behind-the-meter distributed energy resources (DERs), including building solar photovoltaic (PV) technology and electric battery storage, are increasingly being considered as solutions to support carbon reduction goals and increase grid reliability and resiliency. However, dynamic control of these resources in concert with traditional building loads, to effect efficiency and demand flexibility, is not yet commonplace in commercial control products. Traditional rule-based control algorithms do not offer integrated closed-loop control to optimize across systems, and most often, PV and battery systems are operated for energy arbitrage and demand charge management, and not for the provision of grid services. More advanced control approaches, such as MPC control have not been widely adopted in industry because they require significant expertise to develop and deploy. Recent advances in deep reinforcement learning (DRL) offer a promising option to optimize the operation of DER systems and building loads with reduced setup effort. However, there are limited studies that evaluate the efficacy of these methods to control multiple building subsystems simultaneously. Additionally, most of the research has been conducted in simulated environments as opposed to real buildings. This paper proposes a DRL approach that uses a deep deterministic policy gradient algorithm for integrated control of HVAC and electric battery storage systems in the presence of on-site PV generation. The DRL algorithm, trained on synthetic data, was deployed in a physical test building and evaluated against a baseline that uses the current best-in-class rule-based control strategies. Performance in delivering energy efficiency, load shift, and load shed was tested using price-based signals. The results showed that the DRL-based controller can produce cost savings of up to 39.6% as compared to the baseline controller, while maintaining similar thermal comfort in the building. The project team has also integrated the simulation components developed during this work as an OpenAIGym environment and made it publicly available so that prospective DRL researchers can leverage this environment to evaluate alternate DRL algorithms.

电表后分布式能源 (DER)，包括建筑太阳能光伏 (PV) 技术和电池存储，正越来越多地被视为支持碳减排目标和提高电网可靠性和弹性的解决方案。然而，在商业控制产品中，这些资源与传统建筑负载一起动态控制以实现效率和需求灵活性尚不常见。传统的基于规则的控制算法不提供集成的闭环控制来优化跨系统，大多数情况下，光伏和电池系统的运行是为了能源套利和需求充电管理，而不是为了提供电网服务。更先进的控制方法，例如 MPC 控制尚未在工业中广泛采用，因为它们需要大量的专业知识来开发和部署。深度强化学习 (DRL) 的最新进展为优化 DER 系统的运行和构建负载提供了一个有前景的选择，同时减少了设置工作。然而，评估这些方法同时控制多个建筑子系统的功效的研究非常有限。此外，大部分研究都是在模拟环境中进行的，而不是在真实建筑物中进行。本文提出了一种 DRL 方法，该方法使用深度确定性策略梯度算法在存在现场光伏发电的情况下对 HVAC 和电池存储系统进行集成控制。在合成数据上训练的 DRL 算法，部署在物理测试大楼中，并根据使用当前一流的基于规则的控制策略的基线进行评估。使用基于价格的信号测试了在提供能源效率、负载转移和减载方面的性能。结果表明，与基线控制器相比，基于 DRL 的控制器可以节省高达 39.6% 的成本，同时在建筑物中保持类似的热舒适度。项目团队还将在这项工作中开发的模拟组件集成为 OpenAIGym 环境，并将其公开，以便未来的 DRL 研究人员可以利用该环境来评估替代 DRL 算法。并使用基于价格的信号测试减载。结果表明，与基线控制器相比，基于 DRL 的控制器可以节省高达 39.6% 的成本，同时在建筑物中保持类似的热舒适度。项目团队还将在这项工作中开发的模拟组件集成为 OpenAIGym 环境，并将其公开，以便未来的 DRL 研究人员可以利用该环境来评估替代 DRL 算法。并使用基于价格的信号测试减载。结果表明，与基线控制器相比，基于 DRL 的控制器可以节省高达 39.6% 的成本，同时在建筑物中保持类似的热舒适度。项目团队还将在这项工作中开发的模拟组件集成为 OpenAIGym 环境，并将其公开，以便未来的 DRL 研究人员可以利用该环境来评估替代 DRL 算法。