In a cell-free network, a large number of mobile devices are served simultaneously by several base stations (BSs)/access points(APs) using the same time/frequency resources. However, this creates high signal processing demands (e.g., for beamforming) at the transmitters and receivers. In this work, we develop centralized and distributed deep reinforcement learning (DRL)-based methods to optimize beamforming at the uplink of a cell-free network. First, we propose a fully centralized uplink beamforming method (i.e., centralized learning) that uses the Deep Deterministic Policy Gradient algorithm (DDPG) for an offline-trained DRL model. We then enhance this method, in terms of convergence and performance, by using distributed experiences collected from different APs based on the Distributed Distributional Deterministic Policy Gradients algorithm (D4PG) in which the APs represent the distributed agents of the DRL model. To reduce the complexity of signal processing at the central processing unit (CPU), we propose a fully distributed DRL-based uplink beamforming scheme. This scheme divides the beamforming computations among distributed APs. The proposed schemes are then benchmarked against two common linear beamforming schemes, namely, minimum mean square estimation (MMSE) and the simplified conjugate symmetric schemes. The results show that the D4PG scheme with distributed experience achieves the best performance irrespective of the network size. Furthermore, although the proposed distributed beamforming technique reduces the complexity of centralized learning in the DDPG algorithm, it performs better than the DDPG algorithm only for small-scale networks. The performance superiority of the fully centralized DDPG model becomes more evident as the number of APs and/or UEs increases. The codes for all of our DRL implementations are available at https://github.com/RayRedd/Distributed_beamforming_rl.

中文翻译：

---

使用深度强化学习的无细胞无线网络的分布式波束成形技术

在无蜂窝网络中，大量移动设备由多个基站（BS）/接入点（AP）使用相同的时间/频率资源同时服务。然而，这在发射器和接收器处产生了高信号处理需求（例如，对于波束形成）。在这项工作中，我们开发了基于集中式和分布式深度强化学习 (DRL) 的方法来优化无细胞网络上行链路的波束形成。首先，我们提出了一种完全集中的上行链路波束形成方法（即集中学习），该方法将深度确定性策略梯度算法（DDPG）用于离线训练的 DRL 模型。然后，我们在收敛性和性能方面增强了这种方法，通过使用基于分布式分布确定性策略梯度算法 (D4PG) 从不同 AP 收集的分布式经验，其中 AP 表示 DRL 模型的分布式代理。为了降低中央处理单元（CPU）处信号处理的复杂性，我们提出了一种完全分布式的基于 DRL 的上行链路波束成形方案。该方案在分布式 AP 之间划分波束成形计算。然后将所提出的方案与两种常见的线性波束成形方案进行基准测试，即最小均方估计 (MMSE) 和简化的共轭对称方案。结果表明，具有分布式体验的 D4PG 方案无论网络规模如何都能达到最佳性能。此外，尽管所提出的分布式波束成形技术降低了 DDPG 算法中集中学习的复杂性，但它比仅适用于小规模网络的 DDPG 算法表现更好。随着 AP 和/或 UE 数量的增加，完全集中式 DDPG 模型的性能优势变得更加明显。我们所有 DRL 实现的代码都可以在https://github.com/RayRedd/Distributed_beamforming_rl.