Machine learning has begun to play a central role in many applications. A multitude of these applications typically also involve datasets that are distributed across multiple computing devices/machines due to either design constraints or computational/privacy reasons. Such applications often require the learning tasks to be carried out in a decentralized fashion, in which there is no central server that is directly connected to all nodes. In real-world decentralized settings, nodes are prone to undetected failures due to malfunctioning equipment, cyberattacks, etc., which are likely to crash non-robust learning algorithms. The focus of this paper is on robustification of decentralized learning in the presence of nodes that have undergone Byzantine failures. The Byzantine failure model allows faulty nodes to arbitrarily deviate from their intended behaviors, thereby ensuring designs of the most robust of algorithms. But the study of Byzantine resilience within decentralized learning, in contrast to distributed learning, is still in its infancy. In particular, existing Byzantine-resilient decentralized learning methods either do not scale well to large-scale machine learning models, or they lack statistical convergence guarantees that help characterize their generalization errors. In this paper, a scalable, Byzantine-resilient decentralized machine learning framework termed Byzantine-resilient decentralized gradient descent (BRIDGE) is introduced. Algorithmic and statistical convergence guarantees are also provided in the paper for both strongly convex problems and a class of nonconvex problems. In addition, large-scale decentralized learning experiments are used to establish that the BRIDGE framework is scalable and it delivers competitive results for Byzantine-resilient convex and nonconvex learning.

中文翻译：

---

BRIDGE：拜占庭弹性去中心化梯度下降


机器学习已经开始在许多应用中发挥核心作用。由于设计限制或计算/隐私原因，许多这些应用程序通常还涉及分布在多个计算设备/机器上的数据集。此类应用程序通常需要以分散的方式执行学习任务，其中没有直接连接到所有节点的中央服务器。在现实世界的去中心化环境中，节点很容易因设备故障、网络攻击等而出现未被检测到的故障，这可能会导致非鲁棒的学习算法崩溃。本文的重点是在出现拜占庭故障的节点的情况下增强去中心化学习。拜占庭故障模型允许故障节点任意偏离其预期行为，从而确保设计出最稳健的算法。但与分布式学习相比，对去中心化学习中的拜占庭弹性的研究仍处于起步阶段。特别是，现有的拜占庭弹性分散学习方法要么不能很好地扩展到大规模机器学习模型，要么缺乏有助于表征其泛化错误的统计收敛保证。在本文中，介绍了一种可扩展的、具有拜占庭弹性的去中心化机器学习框架，称为拜占庭弹性去中心化梯度下降（BRIDGE）。论文还为强凸问题和一类非凸问题提供了算法和统计收敛保证。 此外，大规模的去中心化学习实验被用来证明 BRIDGE 框架是可扩展的，并且它为拜占庭弹性凸和非凸学习提供了有竞争力的结果。