This paper describes a novel algorithmic framework to minimize a finite-sum of functions available over a network of nodes. The proposed framework, that we call GT-VR, is stochastic and decentralized, and thus is particularly suitable for problems where large-scale, potentially private data, cannot be collected or processed at a centralized server. The GT-VR framework leads to a family of algorithms with two key ingredients: (i) local variance reduction, that enables estimating the local batch gradients from arbitrarily drawn samples of local data; and, (ii) global gradient tracking, which fuses the gradient information across the nodes. Naturally, combining different variance reduction and gradient tracking techniques leads to different algorithms of interest with valuable practical tradeoffs and design considerations. Our focus in this paper is on two instantiations of the ${\bf \mathtt {GT-VR}}$ framework, namely GT-SAGA and GT-SVRG, that, similar to their centralized counterparts (SAGA and SVRG), exhibit a compromise between space and time. We show that both GT-SAGA and GT-SVRG achieve accelerated linear convergence for smooth and strongly convex problems and further describe the regimes in which they achieve non-asymptotic, network-independent linear convergence rates that are faster with respect to the existing decentralized first-order schemes. Moreover, we show that both algorithms achieve a linear speedup in such regimes compared to their centralized counterparts that process all data at a single node. Extensive simulations illustrate the convergence behavior of the corresponding algorithms.

中文翻译：

---

具有加速收敛的方差减少分散随机优化

本文描述了一种新颖的算法框架，以最小化节点网络上可用的函数的有限和。提议的框架，我们称之为 虚拟现实, 是随机和去中心化的，因此特别适用于无法在中央服务器上收集或处理大规模、潜在的私有数据的问题。这虚拟现实 框架导致了一系列具有两个关键要素的算法：(i) 局部方差减少，这可以从任意抽取的本地数据样本中估计本地批次梯度；并且，(ii)全局梯度跟踪，它融合了跨节点的梯度信息。自然地，结合不同的方差减少和梯度跟踪技术会产生不同的感兴趣的算法，并具有有价值的实际权衡和设计考虑。我们在本文中的重点是两个实例 ${\bf \mathtt {GT-VR}}$ 框架，即 GT-SAGA 和 GT-SVRG，即，类似于他们的中心化对手（佐贺 和 SVRG)，表现出空间和时间之间的妥协。我们证明两者 GT-SAGA 和 GT-SVRG实现平滑和强凸问题的加速线性收敛，并进一步描述它们实现非渐近、网络独立线性收敛速率的机制，这些速率相对于现有的分散一阶方案更快。此外，我们表明，与在单个节点处理所有数据的集中式算法相比，这两种算法在这种情况下都实现了线性加速。大量的模拟说明了相应算法的收敛行为。