Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago. The last eight years have seen an exciting new development: variance reduction for stochastic optimization methods. These variance-reduced (VR) methods excel in settings where more than one pass through the training data is allowed, achieving a faster convergence than SGD in theory and practice. These speedups underline the surge of interest in VR methods and the fast-growing body of work on this topic. This review covers the key principles and main developments behind VR methods for optimization with finite data sets and is aimed at nonexpert readers. We focus mainly on the convex setting and leave pointers to readers interested in extensions for minimizing nonconvex functions.

中文翻译：

---

机器学习的方差减少方法

随机优化是机器学习的核心，其基石是随机梯度下降（新元)，这是一种 60 多年前引入的方法。过去八年出现了令人兴奋的新发展：随机优化方法的方差减少。这些方差减少（虚拟现实) 方法在允许多次通过训练数据的设置中表现出色，收敛速度比 新元在理论和实践中。这些加速突显了对虚拟现实方法和有关该主题的快速增长的工作主体。这篇评论涵盖了背后的关键原则和主要发展虚拟现实有限数据集的优化方法，面向非专业读者。我们主要关注凸设置，并将指针留给对最小化非凸函数的扩展感兴趣的读者。