Understanding the evolutionary dynamics of reinforcement learning under multi-agent settings has long remained an open problem. While previous works primarily focus on 2-player games, we consider population games, which model the strategic interactions of a large population comprising small and anonymous agents. This paper presents a formal relation between stochastic processes and the dynamics of independent learning agents who reason based on the reward signals. Using a master equation approach, we provide a novel unified framework for characterising population dynamics via a single partial differential equation (Theorem 1). Through a case study involving Cross learning agents, we illustrate that Theorem 1 allows us to identify qualitatively different evolutionary dynamics, to analyse steady states, and to gain insights into the expected behaviour of a population. In addition, we present extensive experimental results validating that Theorem 1 holds for a variety of learning methods and population games.

中文翻译：

---

人口博弈中自主学习代理的进化动力学

理解多智能体环境下强化学习的进化动力学长期以来一直是一个悬而未决的问题。虽然之前的工作主要关注 2 人游戏，但我们考虑了人口游戏，它模拟了由小型和匿名代理组成的大量人口的战略互动。本文提出了随机过程与基于奖励信号推理的独立学习代理的动态之间的正式关系。使用主方程方法，我们提供了一个新颖的统一框架，用于通过单个偏微分方程（定理 1）来表征种群动态。通过涉及交叉学习代理的案例研究，我们说明定理 1 使我们能够识别定性不同的进化动力学，分析稳态，并深入了解人群的预期行为。此外，我们提供了广泛的实验结果，验证定理 1 适用于各种学习方法和人口游戏。