Skip to main content
SpringerLink
Log in

Riemannian conjugate gradient methods with inverse retraction

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

We propose a new class of Riemannian conjugate gradient (CG) methods, in which inverse retraction is used instead of vector transport for search direction construction. In existing methods, differentiated retraction is often used for vector transport to move the previous search direction to the current tangent space. However, a different perspective is adopted here, motivated by the fact that inverse retraction directly measures the displacement from the current to the previous points in terms of tangent vectors at the current point. The proposed algorithm is implemented with the Fletcher–Reeves and the Dai–Yuan formulae, respectively, and global convergence is established using modifications of the Riemannian Wolfe conditions. Computational details of the practical inverse retractions over the Stiefel and fixed-rank manifolds are discussed. Numerical results obtained for the Brockett cost function minimization problem, the joint diagonalization problem, and the low-rank matrix completion problem demonstrate the potential effectiveness of Riemannian CG with inverse retraction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Absil, P.-A., Baker, C. G., Gallivan, K. A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7, 303–330 (2007)

    Article  MathSciNet  Google Scholar 

  2. Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ (2008)

    Book  Google Scholar 

  3. Absil, P.-A., Malick, J.: Projection-like retractions on matrix manifolds. SIAM J. Optim. 22, 135–158 (2012)

    Article  MathSciNet  Google Scholar 

  4. Absil, P.-A., Oseledets, I. V.: Low-rank retractions: a survey and new results, Comput. Optim. Appl. 62, 5–29 (2015)

    Article  MathSciNet  Google Scholar 

  5. Al-Baali, M.: Descent property and global convergence of the Fletcher-Reeves method with inexact line search. IMA J. Numer. Anal. 5, 121–124 (1985)

    Article  MathSciNet  Google Scholar 

  6. Bonnabel, S.: Stochastic gradient descent on Riemannian manifolds. IEEE Trans. Autom. Control 58, 2217–2229 (2013)

    Article  MathSciNet  Google Scholar 

  7. Boumal, N., Mishra, B., Absil, P.-A., Sepulchre, R.: Manopt, a Matlab toolbox for optimization on manifolds. J. Mach. Learn. Res. 15, 1455–1459 (2014)

    MATH  Google Scholar 

  8. Cardoso, J.F., Souloumiac, A.: Blind beamforming for non-Gaussian signals. IEEE Proceedings F-Radar and Signal Processing 140, 362–370 (1993)

    Article  Google Scholar 

  9. Dai, Y., Yuan, Y.: A nonlinear conjugate gradient method with a strong global convergence property. SIAM J, Optim. 10, 177–182 (1999)

    Article  MathSciNet  Google Scholar 

  10. do Carmo, M. P.: Riemannian Geometry. Translated from the second Portuguese edition by Francis Flaherty. Mathematics: Theory & Applications. Birkh\(\ddot{\rm a}\)user Boston Inc., Boston, MA (1992)

  11. Edelman, A., Arias, T. A., Smith, S. T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)

    Article  MathSciNet  Google Scholar 

  12. Fletcher, R., Reeves, C. M.: Function minimization by conjugate gradients. Comput. J. 7, 149–154 (1964)

    Article  MathSciNet  Google Scholar 

  13. Gallivan, K. A., Qi, C., Absil, P.-A.: A Riemannian Dennis-Moré condition. In: Berry, M.W., Gallivan, K.A., Gallopoulos, E., Grama, A., Philippe, B., Saad, Y., Saied, F. (eds.) High-Performance Scientific Computing, Springer, London, pp. 281–293 (2012)

    Chapter  Google Scholar 

  14. Grohs, P., Hosseini, S.: \(\varepsilon\)-subgradient algorithms for locally Lipschitz functions on Riemannian manifolds. Adv. Comput. Math. 42, 333–360 (2016)

    Article  MathSciNet  Google Scholar 

  15. Hosseini, S., Huang, W., Yousefpour, R.: Line search algorithms for locally Lipschitz functions on Riemannian manifolds. SIAM J. Optim. 28, 596–619 (2018)

    Article  MathSciNet  Google Scholar 

  16. Hosseini, S., Uschmajew, A.: A Riemannian gradient sampling algorithm for nonsmooth optimization on manifolds. SIAM J. Optim. 27, 173–189 (2017)

    Article  MathSciNet  Google Scholar 

  17. Hosseini, S., Uschmajew, A.: A gradient sampling method on algebraic varieties and application to nonsmooth low-rank optimization. SIAM J. Optim. 29, 2853–2880 (2019)

    Article  MathSciNet  Google Scholar 

  18. Hu, J., Liu, X., Wen, Z., Yuan, Y.: A brief introduction to manifold optimization. J. Oper. Res. Soc. China, published online, https://doi.org/10.1007/s40305-020-00295-9

  19. Huang, W.: Optimization algorithms on Riemannian manifolds with applications. Ph.D. thesis, Department of Mathematics, Florida State University (2013)

  20. Huang, W., Absil, P.-A., Gallivan, K. A.: A symmetric rank-one trust-region method. Math. Program. 150, 179–216 (2015)

    Article  MathSciNet  Google Scholar 

  21. Huang, W., Absil, P.-A., Gallivan, K. A.: A Riemannian BFGS method without differentiated retraction for nonconvex optimization problems. SIAM J. Optim. 28, 470–495 (2018)

    Article  MathSciNet  Google Scholar 

  22. Huang, W., Gallivan, K. A., Absil, P.-A.: A Broyden class of quasi-Newton methods for Riemannian optimization. SIAM J. Optim. 25, 1660–1685 (2015)

    Article  MathSciNet  Google Scholar 

  23. Huang, W., Wei, K.: Extending FISTA to Riemannian optimization for sparse PCA. arXiv:1909.05485v1 (2019)

  24. Huang, W., Wei, K.: Riemannnian proximal gradient methods. arXiv:1909.06065v1 (2019)

  25. Jiang, B., Ma, S., Man-Cho So, A., Zhang, S.: Vector transport-free SVRG with general retraction for Riemannian optimization: complexity analysis and practical implementation. arXiv:1705.09059v1 (2017)

  26. Kaneko, T., Fiori, S., Tanaka, T.: Empirical arithmetic averaging over the compact Stiefel manifold. IEEE Transactions on Signal Processing 61, 883–894 (2013)

    Article  MathSciNet  Google Scholar 

  27. Koch, O., Lubich, C.: Dynamical low-rank approximation. SIAM J. Matrix Anal. Appl. 29, 434–454 (2007)

    Article  MathSciNet  Google Scholar 

  28. Lee, J.M.: Introduction to Smooth Manifolds, 2nd edn. Springer, New York, NY (2012)

    Book  Google Scholar 

  29. Nocedal, J., Wright, S. J.: Numerical Optimization, 2nd ed., Springer, New York, NY (2006)

    MATH  Google Scholar 

  30. Qi, C., Gallivan, K. A., Absil, P.-A.: Riemannian BFGS algorithm with applications. In: Recent Advances in Optimization and its Applications in Engineering, Springer-Verlag Berlin, Heidelberg, pp. 183–192 (2010)

  31. Ring, W., Wirth, B.: Optimization methods on Riemannian manifolds and their application to shape space. SIAM J. Optim. 22, 596–627 (2012)

    Article  MathSciNet  Google Scholar 

  32. Rockafellar, R. T., Wets, J.-B.: Variational Analysis. Springer, Berlin, Heidelberg (1998)

    Book  Google Scholar 

  33. Sato, H.: A Dai-Yuan-type Riemannian conjugate gradient method with the weak Wolfe conditions. Comput. Optim. Appl. 64, 101–118 (2016)

    Article  MathSciNet  Google Scholar 

  34. Sato, H., Iwai, T.: A new, globally convergent Riemannian conjugate gradient method. Optimization 64, 1011–1031 (2015)

    Article  MathSciNet  Google Scholar 

  35. Sato, H., Kasai, H., Mishra, B.: Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport. SIAM J. Optim. 29, 1444–1472 (2019)

    Article  MathSciNet  Google Scholar 

  36. Siegel, J. W.: Accelerated optimization with orthogonality constraints. arXiv:1903.05204v3 (2019)

  37. Sun, W., Yuan, Y.: Optimization Theory and Methods: Nonlinear Programming. Springer, Boston, MA (2006)

    MATH  Google Scholar 

  38. Theis, F.J., Cason, T.P., Absil, P.-A.: Soft dimension reduction for ICA by joint diagonalization on the Stiefel manifold. In: Adali, T., Jutten, C., Romano, J.M.T., Barros, A. (eds.) Independent Component Analysis and Signal Separation. Lecture Notes in Computer Science, vol. 5441, pp. 354–361. Springer, Berlin (2009)

    Chapter  Google Scholar 

  39. Tripuraneni, N., Flammarion, N., Bach, F., Jordan, M. I.: Averaging stochastic gradient descent on Riemannian manifolds. PMLR 75, 1–38 (2018)

    Google Scholar 

  40. Vandereycken, B.: Low-rank matrix completion by Riemannian optimization. SIAM J. Optim. 23, 1214–1236 (2013)

    Article  MathSciNet  Google Scholar 

  41. Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Math. Program. 142, 397–434 (2013)

    Article  MathSciNet  Google Scholar 

  42. Zhang, H., Reddi, S. J., Sra, S.: Fast stochastic optimization on Riemannian manifolds. arXiv:1605.07147v1 (2016)

  43. Zhang, H., Sra, S.: First-order methods for geodesically convex optimization. In: JMLR: Workshop and Conference Proceedings vol. 49, pp. 1–22 (2016)

  44. Zhang, H., Sra, S.: An estimate sequence for geodesically convex optimization. PMLR 75, 1–21 (2018)

    Google Scholar 

Download references

Acknowledgements

X. Zhu is supported by National Natural Science Foundation of China Grant Number 11601317 and H. Sato is supported by JSPS KAKENHI Grant Number JP20K14359. The authors are grateful to the coordinating editor and two anonymous referees for their valuable comments and suggestions.

Author information

Authors and Affiliations

  1. College of Mathematics and Physics, Shanghai University of Electric Power, Yangpu District, Shanghai, 200090, China

    Xiaojing Zhu

  2. Department of Applied Mathematics and Physics, Kyoto University, Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan

    Hiroyuki Sato

Authors
  1. Xiaojing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hiroyuki Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaojing Zhu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, X., Sato, H. Riemannian conjugate gradient methods with inverse retraction. Comput Optim Appl 77, 779–810 (2020). https://doi.org/10.1007/s10589-020-00219-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-020-00219-6

Keywords

Search

Navigation