This paper targets developing algorithms for solving distributed learning problems in a communication-efficient fashion, by generalizing the recent method of lazily aggregated gradient (LAG) to deal with stochastic gradient — justifying the name of the new method LASG. While LAG is effective at reducing communication without sacrificing the rate of convergence, we show it only works with deterministic gradients. We introduce new rules and analysis for LASG that are tailored for stochastic gradients, so it effectively saves downloads, uploads, or both for distributed stochastic gradient descent. LASG achieves impressive empirical performance — it typically saves total communication by an order of magnitude. LASG can be used together with gradient quantization to bring more savings.

中文翻译：

---

分布式学习的通信自适应随机梯度方法

本文旨在通过推广最近的惰性聚合梯度 (LAG) 方法来处理随机梯度，从而开发以通信高效的方式解决分布式学习问题的算法——证明新方法 LASG 的名称是合理的。虽然 LAG 在不牺牲收敛速度的情况下有效减少通信，但我们表明它仅适用于确定性梯度。我们为 LASG 引入了为随机梯度量身定制的新规则和分析，因此它有效地为分布式随机梯度下降节省了下载、上传或两者。LASG 实现了令人印象深刻的经验性能——它通常将总通信量减少一个数量级。LASG 可以和梯度量化一起使用，带来更多的节省。