Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, the future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments but also other entities’ policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next-generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in the future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues. Finally, we discuss the challenges, open issues, and future directions related to the applications of MARL in the future Internet.

中文翻译：

---

多智能体强化学习在未来互联网中的应用：综合综述


未来互联网涉及5G及超5G网络、车载网络、无人机（UAV）网络、物联网（IoT）等多种新兴技术。而且，未来的互联网将变得异构化、去中心化，涉及的网络实体数量众多。每个实体可能需要做出本地决策，以在动态和不确定的网络环境下提高网络性能。最近使用单代理强化学习（RL）或深度强化学习（DRL）等标准学习算法，使每个网络实体作为代理能够通过与未知环境交互来自适应地学习最佳决策策略。然而，这种算法无法对网络实体之间的合作或竞争进行建模，并且简单地将其他实体视为环境的一部分，可能导致非平稳性问题。多代理强化学习（MARL）允许每个网络实体不仅通过观察环境而且还通过观察其他实体的策略来学习其最优策略。因此，MARL 可以显着提高网络实体的学习效率，并且最近已被用于解决新兴网络中的各种问题。因此，在本文中，我们回顾了 MARL 在新兴网络中的应用。特别是，我们提供了 MARL 教程以及 MARL 在下一代互联网中的应用的全面调查。特别是，我们首先介绍单代理 RL 和 MARL。然后，我们回顾了 MARL 的一些应用程序，以解决未来互联网中出现的问题。 这些问题包括网络访问、传输功率控制、计算卸载、内容缓存、数据包路由、无人机辅助网络的轨迹设计以及网络安全问题。最后，我们讨论了与 MARL 在未来互联网中的应用相关的挑战、悬而未决的问题和未来方向。