We study preconditioned gradient-based optimization methods where the preconditioning matrix has block-diagonal form. Such a structural constraint comes with the advantage that the update computation can be parallelized across multiple independent tasks. Our main contribution is to demonstrate that the convergence of these methods can significantly be improved by a randomization technique which corresponds to repartitioning coordinates across tasks during the optimization procedure. We provide a theoretical analysis that accurately characterizes the expected convergence gains of repartitioning and validate our findings empirically on various traditional machine learning tasks. From an implementation perspective, block separable models are well suited for parallelization and, when shared memory is available, randomization can be implemented on top of existing methods very efficiently to improve convergence.

中文翻译：

---

并行学习的随机块对角预处理

我们研究了基于预处理梯度的优化方法，其中预处理矩阵具有块对角线形式。这种结构约束的优点是更新计算可以跨多个独立任务并行化。我们的主要贡献是证明这些方法的收敛性可以通过随机化技术显着提高，该技术对应于在优化过程中跨任务重新分配坐标。我们提供了一个理论分析，它准确地描述了重新分区的预期收敛增益，并在各种传统机器学习任务上凭经验验证了我们的发现。从实现的角度来看，块可分离模型非常适合并行化，并且当共享内存可用时，