Abstract
We propose a new way of justifying the accelerated gradient sliding of G. Lan, which allows one to extend the sliding technique to a combination of an accelerated gradient method with an accelerated variance reduction method. New optimal estimates for the solution of the problem of minimizing a sum of smooth strongly convex functions with a smooth regularizer are obtained.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
A. V. Gasnikov, Modern Numerical Optimization Methods: Universal Gradient Descent Method (Mosk. Fiz.-Tekh. Inst., Moscow, 2018) [in Russian].
Z. Allen-Zhu and E. Hazan, Adv. Neural Inf. Process. Syst. 29, 1614–1622 (2016).
A. Beznosikov, E. Gorbunov, and A. Gasnikov, “Derivative-free method for decentralized distributed non-smooth optimization,” The 21st World Congress of the International Federation of Automatic Control (IFAC, 2020).
L. Bottou, F. E. Curtis, and J. Nocedal, SIAM Rev. 60 (2), 223–311 (2018).
P. Dvurechensky, A. Gasnikov, and A. Tiurin, Randomized similar triangles method: A unifying framework for accelerated randomized optimization methods (coordinate descent, directional search, derivative-free method). arXiv:1707.08486
E. Hazan, Lecture Notes: Optimization for Machine Learning. arXiv:1909.03550
A. Ivanova, A. Gasnikov, P. Dvurechensky, D. Dvinskikh, A. Tyurin, E. Vorontsova, and D. Pasechnyuk, Oracle Complexity Separation in Convex Optimization. arXiv:2002.02706
A. Ivanova, D. Grishchenko, A. Gasnikov, and E. Shulgin, Adaptive Catalyst for Smooth Convex Optimization. arXiv:1911.11271
G. Lan, Lectures on Optimization: Methods for Machine Learning. https://pwp.gatech.edu/guanghui-lan/publications/
G. Lan, Z. Li, and Y. Zhou, Adv. Neural Inf. Process. Syst. 32, 10462–10472 (2019).
H. Lin, J. Mairal, and Z. Harchaoui, J. Machine Learning Res. 18 (1), 7854–7907 (2017).
Yu. Nesterov, Math. Program. 140 (1), 125–161 (2013).
Yu. Nesterov, Math. Program. 103 (1), 127–152 (2005).
S. Shalev-Shwartz and S. Ben-David, Understanding Machine Learning: From Theory to Algorithms (Cambridge Univ. Press, Cambridge, 2014).
S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, “Stochastic convex optimization,” COLT (2009).
Funding
This work was supported by the Russian Foundation for Basic Research, project no. 18-31-20005 mol_a_ved (Section 1) and project no. 19-31-90062 Graduate students (Section 2).
Dvinskikh acknowledges the support of the Ministry of Science and Higher Education of the Russian Federation, state assignment no. 075-00337-20-03.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Translated by I. Ruzanova
Rights and permissions
About this article
Cite this article
Dvinskikh, D.M., Omelchenko, S.S., Gasnikov, A.V. et al. Accelerated Gradient Sliding for Minimizing a Sum of Functions. Dokl. Math. 101, 244–246 (2020). https://doi.org/10.1134/S1064562420030084
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1064562420030084