Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. However, given the lack of theoretical insight, it remains unclear what the employed neural networks are learning, or how we should enhance their learning power to address the problems on which they fail. In this work, we empirically investigate the learning power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results extend those in Castellini et al. (Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS’19.International Foundation for Autonomous Agents and Multiagent Systems, pp 1862–1864, 2019) and quantify how well various approaches can represent the requisite value functions, and help us identify the reasons that can impede good performance, like sparsity of the values or too tight coordination requirements.

近年来，深度强化学习技术在合作多智能体系统中的应用取得了巨大的成功。然而，由于缺乏理论洞察力，目前尚不清楚所采用的神经网络正在学习什么，或者我们应该如何增强它们的学习能力来解决它们失败的问题。在这项工作中，我们实证研究了各种网络架构在一系列一次性游戏中的学习能力。尽管它们很简单，但这些游戏捕获了多智能体设置中出现的许多关键问题，例如指数级的联合动作或缺乏明确的协调机制。我们的结果扩展了 Castellini 等人的结果。（第 18 届自治代理和多代理系统国际会议论文集，AAMAS'19。