Several useful variance-reduced stochastic gradient algorithms, such as SVRG, SAGA, Finito, and SAG, have been proposed to minimize empirical risks with linear convergence properties to the exact minimizer. The existing convergence results assume uniform data sampling with replacement. However, it has been observed in related works that random reshuffling can deliver superior performance over uniform sampling and, yet, no formal proofs or guarantees of exact convergence exist for variance-reduced algorithms under random reshuffling. This paper makes two contributions. First, it provides a theoretical guarantee of linear convergence under random reshuffling for SAGA in the mean-square sense; the argument is also adaptable to other variance-reduced algorithms. Second, under random reshuffling, the article proposes a new amortized variance-reduced gradient (AVRG) algorithm with constant storage requirements compared to SAGA and with balanced gradient computations compared to SVRG. AVRG is also shown analytically to converge linearly.

中文翻译：

---

随机重组下的方差减少随机学习

已经提出了几种有用的方差减少的随机梯度算法，例如 SVRG、SAGA、Finito 和 SAG，以通过线性收敛特性将经验风险最小化到精确最小化器。现有的收敛结果假设有替换的统一数据采样。然而，在相关工作中已经观察到，随机重新洗牌可以提供优于均匀采样的性能，但是，对于随机重新洗牌下的方差减少算法，没有正式的证明或精确收敛的保证。本文有两个贡献。首先，它为均方意义上的SAGA在随机reshuffling下线性收敛提供了理论保证；该参数也适用于其他方差减少算法。二、随机改组下，这篇文章提出了一种新的摊销方差减少梯度 (AVRG) 算法，与 SAGA 相比，它具有恒定的存储需求，与 SVRG 相比，具有平衡的梯度计算。AVRG 也被分析显示为线性收敛。