We consider a typical heterogeneous network (HetNet), in which multiple access points (APs) are deployed to serve users by reusing the same spectrum band. Since different APs and users may cause severe interference to each other, advanced power control techniques are needed to manage the interference and enhance the sum-rate of the whole network. Conventional power control techniques first collect instantaneous global channel state information (CSI) and then calculate sub-optimal solutions. Nevertheless, it is challenging to collect instantaneous global CSI in the HetNet, in which global CSI typically changes fast. In this paper, we exploit deep reinforcement learning (DRL) to design a multi-agent power control algorithm in the HetNet. To be specific, by treating each AP as an agent with a local deep neural network (DNN), we propose a multiple-actor-shared-critic (MASC) method to train the local DNNs separately in an online trial-and-error manner. With the proposed algorithm, each AP can independently use the local DNN to control the transmit power with only local observations. Simulations results show that the proposed algorithm outperforms the conventional power control algorithms in terms of both the converged average sum-rate and the computational complexity.

中文翻译：

---

异构网络中多代理功率控制的深度强化学习

我们考虑一个典型的异构网络 (HetNet)，其中部署了多个接入点 (AP)，通过重用相同的频段来为用户提供服务。由于不同的AP和用户之间可能会造成严重的干扰，因此需要先进的功率控制技术来管理干扰并提高整个网络的总速率。传统的功率控制技术首先收集瞬时全局信道状态信息 (CSI)，然后计算次优解。然而，在 HetNet 中收集瞬时全局 CSI 具有挑战性，其中全局 CSI 通常变化很快。在本文中，我们利用深度强化学习 (DRL) 来设计 HetNet 中的多代理功率控制算法。具体来说，通过将每个 AP 视为具有局部深度神经网络 (DNN) 的代理，我们提出了一种多参与者共享评论家 (MASC) 方法，以在线试错方式分别训练本地 DNN。使用所提出的算法，每个 AP 可以独立使用本地 DNN 来控制发射功率，只需本地观察。仿真结果表明，该算法在收敛平均和速率和计算复杂度方面均优于传统的功率控制算法。