Numerical optimization serves as one of the pillars of machine learning. To meet the demands of big data applications, lots of efforts have been put on designing theoretically and practically fast algorithms. This article provides a comprehensive survey on accelerated first-order algorithms with a focus on stochastic algorithms. Specifically, this article starts with reviewing the basic accelerated algorithms on deterministic convex optimization, then concentrates on their extensions to stochastic convex optimization, and at last introduces some recent developments on acceleration for nonconvex optimization.

中文翻译：

---

机器学习的加速一阶优化算法


数值优化是机器学习的支柱之一。为了满足大数据应用的需求，人们在理论和实践快速算法的设计上投入了大量的精力。本文对加速一阶算法进行了全面的调查，重点是随机算法。具体来说，本文首先回顾了确定性凸优化的基本加速算法，然后重点介绍了它们对随机凸优化的扩展，最后介绍了非凸优化加速的一些最新进展。