Skip to main content
SpringerLink
Log in

Local Linear Convergence of the Alternating Direction Method of Multipliers for Nonconvex Separable Optimization Problems

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we consider the convergence rate of the alternating direction method of multipliers for solving the nonconvex separable optimization problems. Based on the error bound condition, we prove that the sequence generated by the alternating direction method of multipliers converges locally to a critical point of the nonconvex optimization problem in a linear convergence rate, and the corresponding sequence of the augmented Lagrangian function value converges in a linear convergence rate. We illustrate the analysis by applying the alternating direction method of multipliers to solving the nonconvex quadratic programming problems with simplex constraint, and comparing it with some state-of-the-art algorithms, the proximal gradient algorithm, the proximal gradient algorithm with extrapolation, and the fast iterative shrinkage–thresholding algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 17–40 (1976)

    Article  Google Scholar 

  2. Glowinski, R., Marrocco, A.: Sur I’approximation par \(\acute{\rm e}\)l\(\acute{\rm e}\)ments finis d’ordre un, et la r\(\acute{\rm e}\)solution, par p\(\acute{\rm e}\)nalisation-dualit\(\acute{\rm e}\) d’une classe de probl\(\grave{\rm e}\)mes de Dirichlet non lin\(\acute{\rm e}\)aires. Revue française d’automatique, informatique, recherche op\(\acute{\rm e}\)rationnelle. Analyse num\(\acute{\rm e}\)rique 9, 41–76 (1975)

  3. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)

    Chapter  Google Scholar 

  4. He, B.S., Yuan, X.M.: On the \(O(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MathSciNet  Google Scholar 

  5. Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23, 475–507 (2013)

    Article  MathSciNet  Google Scholar 

  6. Eckstein, J., Bertsekas, D.P.: An alternating direction method for linear programming. Working paper 90-063, Harvard Business School (1990)

  7. Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM J. Optim. 23, 2183–2207 (2013)

    Article  MathSciNet  Google Scholar 

  8. Han, D.R., Yuan, X.M.: Local linear convergence of the alternating direction method of multipliers for quadratic programs. SIAM J. Numer. Anal. 51, 3446–3457 (2013)

    Article  MathSciNet  Google Scholar 

  9. Deng, W., Yin, W.T.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66, 889–916 (2016)

    Article  MathSciNet  Google Scholar 

  10. Nishihara, R., Lessart, L., Recht, B., Packard, A., Jordan, M.I.: A general analysis of the convergence of ADMM. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, vol. 37, pp. 343–352 (2015)

  11. Yang, W.H., Han, D.R.: Linear convergence of alternating direction method of multipliers for a class of convex optimization problems. SIAM J. Numer. Anal. 54, 625–640 (2016)

    Article  MathSciNet  Google Scholar 

  12. Cai, X.J., Han, D.R., Yuan, X.M.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66, 39–73 (2017)

    Article  MathSciNet  Google Scholar 

  13. Li, G.Y., Pong, T.K.: Global convergence of splitting methods for nonconvex composite optimization. SIAM J. Optim. 25, 2434–2460 (2015)

    Article  MathSciNet  Google Scholar 

  14. Wang, F., Xu, Z.B., Xu, H.K.: Convergence of Bregman alternating direction method with multipliers for nonconvex composite problems. arXiv preprint arXiv:1410.8625 (2014)

  15. Wang, F.H., Cao, W.F., Xu, Z.B.: Convergence of multi-block Bregman ADMM for nonconvex composite problems. Sci. China Inf. Sci. 61, 53–64 (2018)

    MathSciNet  Google Scholar 

  16. Hong, M.Y., Luo, Z.Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim. 26, 337–364 (2016)

    Article  MathSciNet  Google Scholar 

  17. Gonçalves, M.L., Melo, J.G., Monteiro, R.D.: Convergence rate bounds for a proximal ADMM with over-relaxation stepsize parameter for solving nonconvex linearly constrained problems. arXiv preprint arXiv:1702.01850 (2017)

  18. Guo, K., Han, D.R., Wu, T.T.: Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints. Int. J. Comput. Math. 94, 1653–1669 (2017)

    Article  MathSciNet  Google Scholar 

  19. Jiang, B., Lin, T.Y., Ma, S.Q., Zhang, S.Z.: Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis. Comput. Optim. Appl. 72, 115–157 (2019)

    Article  MathSciNet  Google Scholar 

  20. Wang, Y., Yin, W.T., Zeng, J.S.: Global convergence of ADMM in nonconvex nonsmooth optimization. J. Sci. Comput. 78, 29–63 (2019)

    Article  MathSciNet  Google Scholar 

  21. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  22. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka–Łojasiewicz functions and general convergence rates. J. Optim. Theorey Appl. 165, 874–900 (2015)

    Article  Google Scholar 

  23. Li, G.Y., Pong, T.K.: Douglas–Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems. Math. Program. 159, 371–401 (2016)

    Article  MathSciNet  Google Scholar 

  24. Xu, Y.Y., Yin, W.T.: A block coordinate descent method for regularized multi-convex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6, 1758–1789 (2013)

    Article  MathSciNet  Google Scholar 

  25. Wu, Z.M., Li, M., David Wang, Z.W., Han, D.R.: A symmetric alternating direction method of multipliers for separable nonconvex minimization problems. Asia-Pac. J. Oper. Res. 34, 1–27 (2017)

    MathSciNet  Google Scholar 

  26. Luo, Z.Q., Tseng, P.: Error bound and convergence analysis of matrix splitting algorithms for the affine variational inequality problem. SIAM J. Optim. 2, 43–54 (1992)

    Article  MathSciNet  Google Scholar 

  27. Luo, Z.Q., Tseng, P.: Error bounds and convergence analysis of feasible descent methods: a general approach. Ann. Oper. Res. 46, 157–178 (1993)

    Article  MathSciNet  Google Scholar 

  28. Attouch, H., Bolte, J., Svaiter, B.F., Soubeyran, A.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137, 91–129 (2013)

    Article  MathSciNet  Google Scholar 

  29. Ibaraki, T., Katoh, N.: Resource Allocation Problems: Algorithmic Approaches. MIT Press, Cambridge (1988)

    Google Scholar 

  30. Markowitz, H.: Portfolio selection. J. Finance 7, 77–91 (1952)

    Google Scholar 

  31. Chen, X., Peng, J.M., Zhang, S.Z.: Sparse solutions to random standard quadratic optimization problems. Math. Program. 141, 273–293 (2013)

    Article  MathSciNet  Google Scholar 

  32. Gibbons, L.E., Hearn, D.W., Pardalos, P.M., Ramana, M.V.: Continuous characterizations of the maximum clique problem. Math. Oper. Res. 22, 754–768 (1997)

    Article  MathSciNet  Google Scholar 

  33. Bomze, I.M., Locatelli, M., Tardella, F.: New and old bounds for standard quadratic optimization: dominance, equivalence and incomparability. Math. Program. 115, 31–64 (2008)

    Article  MathSciNet  Google Scholar 

  34. Kaplan, W.: A test for copositive matrices. Linear Algebra Appl. 313, 203–206 (2000)

    Article  MathSciNet  Google Scholar 

  35. Quadratic programming criteria for copositive matrices: V\(\ddot{\rm a}\)liaho H. Linear Algebra Appl. 119, 163–182 (1989)

  36. Xu, Z.B., Chang, X.Y., Xu, F.M., Zhang, H.: \( L_ {1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23, 1013–1027 (2012)

    Article  Google Scholar 

  37. Liu, J., Chen, J.H., Ye, J.P.: Large-scale sparse logistic regression. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 547–556. ACM (2009)

  38. Parikh, N., Boyd, S.: Proximal algorithms. Founda. Trends Optim. 1, 127–239 (2014)

    Article  Google Scholar 

  39. Wen, B., Chen, X.J., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27, 124–145 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This research was supported by National Natural Science Foundation of China (Grant Nos. 11625105, 11431002, 11801279, 11871279, and 11571178), Natural Science Foundation of Jiangsu Province (Grant No. BK20180782), and the Startup Foundation for Introducing Talent of NUIST (Grant No. 2017r059).

Author information

Authors and Affiliations

  1. Department of Information and Computing Science, School of Mathematics and Statistics, Nanjing University of Information Science and Technology, Nanjing, 210044, People’s Republic of China

    Zehui Jia

  2. School of Mathematical Sciences, Key Laboratory for NSLSCS of Jiangsu Province, Nanjing Normal University, Nanjing, 210023, People’s Republic of China

    Xue Gao, Xingju Cai & Deren Han

  3. School of Mathematical Sciences, Beijing Advanced Innovation Center for Big Data and Brain Computing (BDBC), Beihang University, Beijing, 100191, People’s Republic of China

    Deren Han

Authors
  1. Zehui Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xue Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingju Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deren Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deren Han.

Additional information

Communicated by Hedy Attouch.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, Z., Gao, X., Cai, X. et al. Local Linear Convergence of the Alternating Direction Method of Multipliers for Nonconvex Separable Optimization Problems. J Optim Theory Appl 188, 1–25 (2021). https://doi.org/10.1007/s10957-020-01782-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-020-01782-y

Keywords

Mathematics Subject Classification

Search

Navigation