Abstract
We consider the problem of minimization for a function with Lipschitz continuous gradient on a proximally smooth and smooth manifold in a finite dimensional Euclidean space. We consider the Lezanski-Polyak-Lojasiewicz (LPL) conditions in this problem of constrained optimization. We prove that the gradient projection algorithm for the problem converges with a linear rate when the LPL condition holds.
Similar content being viewed by others
References
Goldstein, A.A.: Convex programming in Hilbert space. Bull. Amer. Math. Soc. 70:5, 709–710 (1964)
Levitin, E.S., Polyak, B.T.: Constrained minimization methods. Zh. Vychisl. Mat. Mat. Fiz. 6:5, 787–823 (1966)
Polyak, B.T.: Introduction to Optimization. M., Science (1983)
Nesterov, Y.u.: Introductory Lectures on Convex Optimization. Springer, Abasic course basic course (2004)
Karimi, H., Nutini, J., Schmidt, M.: Linear Convergence of Gradient and Proximal-Gradient Methods under the Polyak-Lojasiewicz Condition Linear Convergence of Gradient and 1 Methods under the 2 Condition. In: Frasconi, P., Landwehr, N., Manco, G., Vreeken, J (eds.) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2016. Lecture Notes in Computer Science, vol. 9851. Springer, Cham (2016)
Ležanski, T.: ÜBer das Minimumproblem für Funktionale in Banachschen räumen. Math. Ann. 152, 271–274 (1963)
Polyak, B.T.: Gradient methods for minimizing functionals. Zh. Vychisl. Mat. Mat. Fiz. 3:4, 643–653 (1963)
Lojasiewicz, S.: A Topological Property of Real Analytic Subsets (in French). Coll. du CNRS. Les Equations aux derives partielles 117, 87–89 (1963)
Lojasiewicz, S.: Sur le problem de la division. Studia Math. 8, 87–136 (1959)
A.D: Ioffe METRIC REGULARITY A SURVEY arXiv:1505.07920v2 (2015)
Luo, Z.-Q.: New error bounds and their applications to convergence analysis of iterative algorithms. Math. Program. Ser. B. V. 88, 341–355 (2000)
Maxim, V., Maxim, O.: Golubev Balashov About the Lipschitz property of the metric projection in the Hilbert space. J. Math. Anal. Appl. 394:2, 545–551 (2012)
Balashov, M.V.: Maximization of a function with Lipschitz continuous gradient. J. Math. Sci. 209:1, 12–18 (2015)
Drusvyatskiy, D., Lewis, A.: Error bounds, quadratic growth, and linear convergence of proximal methods arXiv:1602.06661v2 (2016)
Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20:2, 303–353 (1998)
Luenberger, D.G.: The gradient projection methods along geodesics. Manag. Sci. 18:11, 620–631 (1972)
Hager, W.W.: Minimizing a quadratic over a sphere. SIAM J. Optim. Contr. 12:1, 188–208 (2001)
Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization algorithms on matrix manifolds. Princeton University Press, Princeton and Oxford (2008)
da Cruz Neto, J.X., De Lima, L.L., Oliveira, P.R.: Geodesic algorithms on Riemannian manifolds. Balkan J. Geom. Appl. 3:2, 89–100 (1998)
Udrişte, C.: Convex Functions and Optimization Methods on Riemannian Manifolds. Mathematics and Its Applications series, vol. 297. Springer (1994)
Schneider, R., Uschmajew, A.: Convergence results for projected line search methods on varieties of low-rank matricies via Lojasiewicz inequality, vol. 25:1 (2015)
Jain, P., Kar, P.: Non-convex Optimization for Machine Learning. Now Foundations and Trends, pp. 154 (2017)
Barber, R. F., Ha, W.: Gradient descent with nonconvex constraints: local concavity determines convergence, arXiv:1703.07755v3 (2017)
Balashov, M., Polyak, B., Tremba, A.: Gradient projection and conditional gradient methods for constrained nonconvex minimization. Numerical Functional Analysis and Optimization 41(7), 822–849 (2020)
Absil, P.-A., Malick, J.: Projection-like retraction on matrix manifolds. SIAM J. Optim. 22:1, 135–158 (2012)
Ivanov, G.E.: Weakly convex sets and functions, M., Fizmatlit. In Russian (2006)
Vial, J.-P.h.: Strong and weak convexity of sets and functions. Math. Oper. Res. 8:2, 231–259 (1983)
Clarke, F.H., Stern, R.J., Wolenski, P.R.: Proximal smoothness and lower–C2 property. J. Convex Anal. 2:1-2, 117–144 (1995)
Balashov, M.V.: The gradient projection algorithm for a proximally smooth set and a function with lipschitz continuous gradient. Sbornik: Mathematics 211(4), 481–504 (2020)
Balashov, M.V., Ivanov, G.E.: Weakly convex and proximally smooth sets in Banach spaces. Izv. RAN. Ser. Mat. 73:3, 23–66 (2009)
Goncharov, V.V., Ivanov, G.E.: Strong and Weak Convexity of Closed Sets in a Hilbert Space. In: Daras, N., Rassias, T. (eds.) Operations Research, Engineering, and Cyber Security. Springer Optimization and Its Applications, vol. 113, pp 259–297. Springer, Cham (2017)
Bounkhel, M., Thibault, L.: On various notions of regularity of sets in nonsmooth analysis. Nonlin. Anal. 48, 223–246 (2002)
Poliquin, R.A., Rockafellar, R.T., Thibault, L.: Local differentiability of distance functions. Trance. Amer. Math. Soc. 353, 5231–5249 (2000)
Balashov, M.V.: About the gradient projection algorithm for a strongly convex function and a proximally smooth set. J. Convex Anal. 24:2, 493–500 (2017)
Gao, B., Liu, X., Chen, X., Yuan, Y.-X.: On the Lojasiewicz Exponent of the Quadratic Sphere Constrained Optimization Problem, arXiv:1611.08781v2
Liu, H., Wu, W., So, A.M.C.: Quadratic Optimization with Orthogonality Constraints:Explicit Lojasiewicz Exponent and Linear Convergence of Line-Search Methods, arXive:1510.01025v1 (2015)
Zhang, J., Zhang, S.: A Cubic Regularized Newton’s Method over Riemannian Manifolds, arXive:1805.05565v1 (2018)
Merlet, B., Nguyen, T.N.: Convergence to equilibrium for discretizations of gradient-like flows on Riemannian manifolds, vol. 26 (2013)
Acknowledgments
The author is greatful to B. T. Polyak for useful comments.
The author is grateful to the reviewers for numerous comments and suggestions.
The work was supported by Russian Science Foundation (Project 16-11-10015).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Balashov, M.V. The Gradient Projection Algorithm for Smooth Sets and Functions in Nonconvex Case. Set-Valued Var. Anal 29, 341–360 (2021). https://doi.org/10.1007/s11228-020-00550-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11228-020-00550-4
Keywords
- Lipschitz continuous gradient
- Proximal smoothness
- Gradient projection algorithm
- Metric projection
- Nonconvex extremal problem
- Lezanski-Polyak-Lojasiewicz condition