While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

中文翻译：

---

具有补偿重叠的高效通信联邦学习-FedAvg


虽然许多独立的计算设备每天都会产生数PB的数据，但由于数据安全和隐私泄露的担忧，最终只有少数数据能够被收集并用于深度学习（DL），从而严重阻碍了数据的扩展。 DL。在这种情况下，联邦学习（FL）被提出，通过多个客户端的组合数据进行模型训练，而无需在集群内共享数据集。然而，基于周期性模型平均的联邦学习（FedAvg）引入了大量的通信开销，因为每次迭代中的同步数据与模型大小大致相同，从而导致通信效率较低。因此，提出了侧重于通信轮次减少和数据压缩的变体提案，以减少 FL 的通信开销。在本文中，我们提出了 Overlap-FedAvg，这是一种创新框架，它放松了联邦学习的链式约束，并将模型训练阶段与模型通信阶段（即上传本地模型和下载全局模型）并行化，从而使后一阶段可以完全被前一阶段覆盖。与普通的 FedAvg 相比，Overlap-FedAvg 进一步发展了分层计算策略、数据补偿机制和 Nesterov 加速梯度（NAG）算法。特别是，Overlap-FedAvg 与许多其他压缩方法正交，因此它们可以一起应用以最大限度地提高集群的利用率。此外，还提供了理论分析来证明所提出框架的收敛性。 使用多个模型和数据集对图像分类和自然语言处理任务进行的大量实验也表明，所提出的框架大大减少了通信开销并促进了联邦学习过程。