Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the involved algorithms. Second, we introduce an approximate variant of the gradient coding problem, in which we settle for approximate gradient computation instead of the exact one. This approach enables graceful degradation, i.e., the $\ell _{2}$ error of the approximate gradient is a decreasing function of the number of stragglers. Our main result is that normalized adjacency matrices of expander graphs yield excellent approximate gradient codes, which enable significantly less computation compared to exact gradient coding, and guarantee faster convergence than trivial solutions under standard assumptions. We experimentally test our approach on Amazon EC2, and show that the generalization error of approximate gradient coding is very close to the full gradient while requiring significantly less computation from the workers.

中文翻译：

---

循环 MDS 代码和扩展图的梯度编码

梯度编码是一种在分布式学习中缓解掉队现象的技术。在本文中，我们使用经典编码理论中的工具设计了新颖的梯度码，即循环 MDS 码，无论是在参数适用范围还是在所涉及算法的复杂性方面，它都与现有解决方案相比具有优势。其次，我们介绍了梯度编码问题的一种近似变体，其中我们解决了近似梯度计算而不是精确梯度计算。这种方法可以实现优雅的退化，即近似梯度的 $\ell_{2}$ 误差是落后者数量的递减函数。我们的主要结果是扩展图的归一化邻接矩阵产生了极好的近似梯度代码，与精确梯度编码相比，这可以显着减少计算量，并保证比标准假设下的平凡解决方案更快的收敛。我们在 Amazon EC2 上对我们的方法进行了实验测试，结果表明近似梯度编码的泛化误差非常接近全梯度，同时需要工作人员的计算量显着减少。