One major challenge in water resource management is to balance the uncertain and nonstationary water demands and supplies caused by the changing anthropogenic and hydroclimate conditions. To address this issue, we developed a reinforcement learning agent-based modeling (RL-ABM) framework where agents (agriculture water users) are able to learn and adjust water demands based on their interactions with the water systems. The intelligent agents are created by a reinforcement learning algorithm adapted from the Q-learning algorithm. We illustrated this framework in a case study where the RL-ABM is two-way coupled with the Colorado River Simulation System (CRSS), a long-term planning model used for the administration of the Colorado River Basin, for assessing agriculture water uses impacts on water scarcity. Seventy-eight intelligent agents are simulated, which can be grouped into three categories based on their parameter values: the “aggressive” (swift actions; low regrets), the “forward-looking conservative” (mild actions; high regrets; fast learning), and the “myopic conservative” (mild actions; median regrets; slow learning). The ABM-CRSS results showed that the major reservoirs in the Upper Colorado Basin might experience more frequent water shortages due to the increasing water uses compared to the original CRSS results. If the drought continues, the case study also demonstrates that agents can learn and adjust their demands.

中文翻译：

---

评估适应性灌溉对非平稳环境中水资源短缺的影响——一种多代理强化学习方法

水资源管理的一项主要挑战是平衡由不断变化的人为和水文气候条件引起的不确定和不稳定的水需求和供应。为了解决这个问题，我们开发了一个基于强化学习代理的建模 (RL-ABM) 框架，其中代理（农业用水用户）能够根据与水系统的相互作用来学习和调整用水需求。智能代理由从 Q-learning 算法改编而来的强化学习算法创建。我们在一个案例研究中说明了这个框架，其中 RL-ABM 与科罗拉多河模拟系统 (CRSS) 是双向耦合的，这是一种用于科罗拉多河流域管理的长期规划模型，用于评估农业用水影响关于水资源短缺。模拟了 78 个智能代理，根据它们的参数值可以分为三类：“积极”（快速行动；低后悔），“前瞻性保守”（温和行动；高后悔；快速学习） ，以及“近视保守派”（温和的行动；中等后悔；缓慢的学习）。ABM-CRSS 结果显示，与原始 CRSS 结果相比，由于用水量增加，上科罗拉多盆地的主要水库可能会更频繁地缺水。如果干旱持续，案例研究还表明代理可以学习和调整他们的需求。和“近视保守派”（温和的行动；中等后悔；缓慢的学习）。ABM-CRSS 结果显示，与原始 CRSS 结果相比，由于用水量增加，上科罗拉多盆地的主要水库可能会更频繁地缺水。如果干旱持续，案例研究还表明代理可以学习和调整他们的需求。和“近视保守派”（温和的行动；中等后悔；缓慢的学习）。ABM-CRSS 结果显示，与原始 CRSS 结果相比，由于用水量增加，上科罗拉多盆地的主要水库可能会更频繁地缺水。如果干旱持续，案例研究还表明代理可以学习和调整他们的需求。