Abstract A new generation of smart stormwater systems promises to reduce the need for new construction by enhancing the performance of the existing infrastructure through real-time control. Smart stormwater systems dynamically adapt their response to individual storms by controlling distributed assets, such as valves, gates, and pumps. This paper introduces a real-time control approach based on Reinforcement Learning (RL), which has emerged as a state-of-the-art methodology for autonomous control in the artificial intelligence community. Using a Deep Neural Network, a RL-based controller learns a control strategy by interacting with the system it controls - effectively trying various control strategies until converging on those that achieve a desired objective. This paper formulates and implements a RL algorithm for the real-time control of urban stormwater systems. This algorithm trains a RL agent to control valves in a distributed stormwater system across thousands of simulated storm scenarios, seeking to achieve water level and flow set-points in the system. The algorithm is first evaluated for the control of an individual stormwater basin, after which it is adapted to the control of multiple basins in a larger watershed (4 km2). The results indicate that RL can very effectively control individual sites. Performance is highly sensitive to the reward formulation of the RL agent. Generally, more explicit guidance led to better control performance, and more rapid and stable convergence of the learning process. While the control of multiple distributed sites also shows promise in reducing flooding and peak flows, the complexity of controlling larger systems comes with a number of caveats. The RL controller’s performance is very sensitive to the formulation of the Deep Neural Network and requires a significant amount of computational resource to achieve a reasonable performance enhancement. Overall, the controlled system significantly outperforms the uncontrolled system, especially across storms of high intensity and duration. A frank discussion is provided, which should allow the benefits and drawbacks of RL to be considered when implementing it for the real-time control of stormwater systems. An open source implementation of the full simulation environment and control algorithms is also provided.

中文翻译：

---

实时控制雨水系统的深度强化学习

摘要 新一代智能雨水系统有望通过实时控制提高现有基础设施的性能，从而减少对新建筑的需求。智能雨水系统通过控制分布式资产（例如阀门、闸门和泵）来动态调整对单个风暴的响应。本文介绍了一种基于强化学习 (RL) 的实时控制方法，该方法已成为人工智能社区中最先进的自主控制方法。使用深度神经网络，基于 RL 的控制器通过与其控制的系统交互来学习控制策略 - 有效地尝试各种控制策略，直到收敛于那些实现预期目标的策略。本文制定并实现了一种用于城市雨水系统实时控制的 RL 算法。该算法训练 RL 代理在数千个模拟风暴场景中控制分布式雨水系统中的阀门，寻求达到系统中的水位和流量设定点。该算法首先针对单个雨水流域的控制进行评估，然后适用于更大流域 (4 km2) 中多个流域的控制。结果表明 RL 可以非常有效地控制单个站点。性能对 RL 代理的奖励公式高度敏感。一般来说，更明确的指导导致更好的控制性能，以及更快速和更稳定的学习过程收敛。虽然对多个分布式站点的控制也显示出减少洪水和峰值流量的希望，但控制大型系统的复杂性伴随着许多注意事项。RL 控制器的性能对深度神经网络的公式非常敏感，需要大量的计算资源才能实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，尤其是在高强度和持续时间的风暴中。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。控制大型系统的复杂性伴随着许多注意事项。RL 控制器的性能对深度神经网络的公式非常敏感，需要大量的计算资源才能实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，尤其是在高强度和持续时间的风暴中。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。控制大型系统的复杂性伴随着许多注意事项。RL 控制器的性能对深度神经网络的公式非常敏感，需要大量的计算资源才能实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，尤其是在高强度和持续时间的风暴中。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。RL 控制器的性能对深度神经网络的公式非常敏感，需要大量的计算资源才能实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，尤其是在高强度和持续时间的风暴中。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。RL 控制器的性能对深度神经网络的公式非常敏感，需要大量的计算资源才能实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，尤其是在高强度和持续时间的风暴中。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。提供了一个坦率的讨论，这应该允许在实施 RL 以实时控制雨水系统时考虑 RL 的优点和缺点。还提供了完整仿真环境和控制算法的开源实现。