Abstract
As is well known, when initialized close to a nonsingular solution of a smooth nonlinear equation, the Newton method converges to this solution superlinearly. Moreover, the common Armijo linesearch procedure used to globalize the process for convergence from arbitrary starting points, accepts the unit stepsize asymptotically and ensures fast local convergence. In the case of a singular and possibly even nonisolated solution, the situation is much more complicated. Local linear convergence (with asymptotic ratio of 1/2) of the Newton method can still be guaranteed under reasonable assumptions, from a starlike, asymptotically dense set around the solution. Moreover, convergence can be accelerated by extrapolation and overrelaxation techniques. However, nothing was previously known on how the Newton method can be coupled in these circumstances with a linesearch technique for globalization that locally accepts unit stepsize and guarantees linear convergence. It turns out that this is a rather nontrivial issue, requiring a delicate combination of the analyses on acceptance of the unit stepsize and on the iterates staying within the relevant starlike domain of convergence. In addition to these analyses, numerical illustrations and comparisons are presented for the Newton method and the use of extrapolation to accelerate convergence speed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Facchinei, F., Fischer, A., Herrich, M.: A family of Newton methods for nonsmooth constrained systems with nonisolated solutions. Math. Methods Oper. Res. 77, 433–443 (2013)
Facchinei, F., Fischer, A., Herrich, M.: An LP-Newton method: nonsmooth equations, KKT systems, and nonisolated solutions. Math. Program. 146, 1–36 (2014)
Fernández, D., Solodov, M.: Stabilized sequential quadratic programming for optimization and a stabilized Newton-type method for variational problems. Math. Program. 125, 47–73 (2010)
Fischer, A., Izmailov, A.F., Solodov, M.V.: Local attractors of Newton-type methods for constrained equations and complementarity problems with nonisolated solutions. J. Optim. Theory Appl. 180, 140–169 (2019)
Fan, J.-Y., Yuan, Y.-X.: On the quadratic convergence of the Levenberg–Marquardt method. Computing 74, 23–39 (2005)
Griewank, A.: Analysis and modification of Newton’s method at singularities. Ph.D. thesis. Australian National University, Canberra (1980)
Griewank, A.: Starlike domains of convergence for Newton’s method at singularities. Numer. Math. 35, 95–111 (1980)
Griewank, A.: On solving nonlinear equations with simple singularities or nearly singular solutions. SIAM Rev. 27, 537–563 (1985)
Izmailov, A.F., Kurennoy, A.S., Solodov, M.V.: Critical solutions of nonlinear equations: local attraction for Newton-type methods. Math. Program. 167, 355–379 (2018)
Izmailov, A.F., Kurennoy, A.S., Solodov, M.V.: Critical solutions of nonlinear equations: stability issues. Math. Program. 168, 475–507 (2018)
Izmailov, A.F., Solodov, M.V.: Stabilized SQP revisited. Math. Program. 122, 93–120 (2012)
Izmailov, A.F., Solodov, M.V.: Newton-Type Methods for Optimization and Variational Problems. Springer Series in Operations Research and Financial Engineering. Springer, Cham (2014)
Izmailov, A.F., Tretyakov, A.A.: \(2\)-Regular Solutions of Nonlinear Problems. Theory and Numerical Methods. Fizmatlit, Moscow (1999). (in Russian)
Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)
Marquardt, D.W.: An algorithm for least squares estimation of non-linear parameters. SIAM J. 11, 431–441 (1963)
Oberlin, C., Wright, S.J.: An accelerated Newton method for equations with semismooth Jacobians and nonlinear complementarity problems. Math. Program. 117, 355–386 (2009)
Wright, S.J.: Superlinear convergence of a stabilized SQP method to a degenerate solution. Comput. Optim. Appl. 11, 253–275 (1998)
Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. Comput. Suppl. 15, 237–249 (2001)
Acknowledgements
The authors thank Ivan Rodin for his assistance with numerical experiments, and the two anonymous referees for their useful comments on the original version of this article.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The first author is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—409756759. The second author is supported in part by the Russian Foundation for Basic Research Grants 19-51-12003 NNIO_a and 20-01-00106, and by Volkswagen Foundation. The third author is supported by CNPq Grant 303724/2015-3, by FAPERJ Grant E-26/202.540/2019, and by PRONEX–Optimization.
Rights and permissions
About this article
Cite this article
Fischer, A., Izmailov, A.F. & Solodov, M.V. Unit stepsize for the Newton method close to critical solutions. Math. Program. 187, 697–721 (2021). https://doi.org/10.1007/s10107-020-01496-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-020-01496-z
Keywords
- Nonlinear equation
- Newton method
- Singular solution
- Critical solution
- 2-Regularity
- Linear convergence
- Superlinear convergence
- Extrapolation
- Linesearch