We develop a new approach to tackle communication constraints in a distributed learning problem with a central server. We propose and analyze a new algorithm that performs bidirectional compression and achieves the same convergence rate as algorithms using only uplink (from the local workers to the central server) compression. To obtain this improvement, we design MCM, an algorithm such that the downlink compression only impacts local models, while the global model is preserved. As a result, and contrary to previous works, the gradients on local servers are computed on perturbed models. Consequently, convergence proofs are more challenging and require a precise control of this perturbation. To ensure it, MCM additionally combines model compression with a memory mechanism. This analysis opens new doors, e.g. incorporating worker dependent randomized-models and partial participation.

中文翻译：

---

保留中央模型，可在分布式设置中更快地进行双向压缩

我们开发了一种新方法来解决中央服务器在分布式学习问题中的通信限制。我们提出并分析了一种新的算法，该算法执行双向压缩并获得与仅使用上行链路（从本地工作人员到中央服务器）压缩的算法相同的收敛速度。为了获得这种改进，我们设计了MCM算法，该算法使得下行链路压缩仅影响局部模型，而保留了全局模型。结果，与先前的工作相反，本地服务器上的梯度是在扰动模型上计算的。因此，收敛证明更具挑战性，并且需要对此扰动进行精确控制。为了确保这一点，MCM还将模型压缩与存储机制结合在一起。这种分析打开了新的大门，例如