The alternating direction method of multipliers (ADMM) has been recently recognized as a promising optimizer for large-scale machine learning models. However, there are very few results studying ADMM from the aspect of communication costs, especially jointly with privacy preservation, which are critical for distributed learning. We investigate the communication efficiency and privacy-preservation of ADMM by solving the consensus optimization problem over decentralized networks. Since walk algorithms can reduce communication load, we first propose incremental ADMM (I-ADMM) based on the walk algorithm, the updating order of which follows a Hamiltonian cycle instead. However, I-ADMM cannot guarantee the privacy for agents against external eavesdroppers even if the randomized initialization is applied. To protect privacy for agents, we then propose two privacy-preserving incremental ADMM algorithms, i.e., PI-ADMM1 and PI-ADMM2, where perturbation over step sizes and primal variables is adopted, respectively. Through theoretical analyses, we prove the convergence and privacy preservation for PI-ADMM1, which are further supported by numerical experiments. Besides, simulations demonstrate that the proposed PI-ADMM1 and PI-ADMM2 algorithms are communication efficient compared with state-of-the-art methods.

中文翻译：

---

用于分散共识优化的隐私保护增量 ADMM

乘法器的交替方向法（ADMM）最近被认为是大规模机器学习模型的有前途的优化器。然而，很少有从通信成本方面研究 ADMM 的结果，特别是与隐私保护相结合，这对于分布式学习至关重要。我们通过解决分散网络上的共识优化问题来研究 ADMM 的通信效率和隐私保护。由于walk算法可以减少通信负载，我们首先在walk算法的基础上提出增量ADMM（I-ADMM），其更新顺序遵循哈密顿循环。然而，即使应用了随机初始化，I-ADMM 也不能保证代理的隐私不受外部窃听者的影响。为了保护代理人的隐私，然后，我们提出了两种隐私保护增量 ADMM 算法，即 PI-ADMM1 和 PI-ADMM2，其中分别采用对步长和原始变量的扰动。通过理论分析，我们证明了 PI-ADMM1 的收敛性和隐私保护，并得到了数值实验的进一步支持。此外，仿真表明，与最先进的方法相比，所提出的 PI-ADMM1 和 PI-ADMM2 算法的通信效率更高。