Skip to main content
SpringerLink
Log in

Single-forward-step projective splitting: exploiting cocoercivity

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

This work describes a new variant of projective splitting for solving maximal monotone inclusions and complicated convex optimization problems. In the new version, cocoercive operators can be processed with a single forward step per iteration. In the convex optimization context, cocoercivity is equivalent to Lipschitz differentiability. Prior forward-step versions of projective splitting did not fully exploit cocoercivity and required two forward steps per iteration for such operators. Our new single-forward-step method establishes a symmetry between projective splitting algorithms, the classical forward–backward splitting method (FB), and Tseng’s forward-backward-forward method. The new procedure allows for larger stepsizes for cocoercive operators: the stepsize bound is \(2\beta\) for a \(\beta\)-cocoercive operator, the same bound as has been established for FB. We show that FB corresponds to an unattainable boundary case of the parameters in the new procedure. Unlike FB, the new method allows for a backtracking procedure when the cocoercivity constant is unknown. Proving convergence of the algorithm requires some departures from the prior proof framework for projective splitting. We close with some computational tests establishing competitive performance for the method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. If backtracking is used, then all three of these methods can converge under weaker local continuity assumptions.

  2. The breast cancer dataset is available at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE1379. The IBD dataset is available at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE3365.

  3. The C1 dataset is available at http://software.broadinstitute.org/gsea/index.jsp.

  4. Overlapping group norms can also be handled with our method, but using a different problem formulation than (57).

  5. TripAdvisor data is available at https://github.com/yanxht/TripAdvisorData or through our repository at https://github.com/projective-splitting/coco.

References

  1. Alotaibi, A., Combettes, P.L., Shahzad, N.: Solving coupled composite monotone inclusions by successive Fejér approximations of their Kuhn–Tucker set. SIAM J. Optim. 24(4), 2076–2095 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  2. Baillon, J.B., Haddad, G.: Quelques propriétés des opérateurs angle-bornés \(n\)-cycliquement monotones. Isr. J. Math. 26(2), 137–150 (1977)

    MATH  Google Scholar 

  3. Bauschke, H.H., Combettes, P.L.: The Baillon-Haddad theorem revisited. J. Convex Anal. 17(3–4, SI), 781–787 (2010)

    MathSciNet  MATH  Google Scholar 

  4. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd edn. Springer, Berlin (2017)

    Book  MATH  Google Scholar 

  5. Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Burczynski, M.E., Peterson, R.L., Twine, N.C., Zuberek, K.A., Brodeur, B.J., Casciotti, L., Maganti, V., Reddy, P.S., Strahs, A., Immermann, F., et al.: Molecular classification of Crohn’s disease and ulcerative colitis patients using transcriptional profiles in peripheral blood mononuclear cells. J. Mol. Diagn. 8(1), 51–61 (2006)

    Article  Google Scholar 

  7. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Combettes, P.L.: Systems of structured monotone inclusions: duality, algorithms, and applications. SIAM J. Optim. 23(4), 2420–2447 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  9. Combettes, P.L., Eckstein, J.: Asynchronous block-iterative primal-dual decomposition methods for monotone inclusions. Math. Program. 168(1–2), 645–672 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  10. Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing. In: Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer, Berlin (2011)

  11. Combettes, P.L., Pesquet, J.C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set-Valued Var. Anal. 20(2), 307–330 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  12. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  13. Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set-Valued Var. Anal. 25(4), 829–858 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  14. Diamond, S., Boyd, S.: CVXPY: a Python-embedded modeling language for convex optimization. J. Mach. Learn. Res. 17(83), 1–5 (2016)

    MathSciNet  MATH  Google Scholar 

  15. Eckstein, J.: A simplified form of block-iterative operator splitting and an asynchronous algorithm resembling the multi-block alternating direction method of multipliers. J. Optim. Theory Appl. 173(1), 155–182 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Eckstein, J., Svaiter, B.F.: A family of projective splitting methods for the sum of two maximal monotone operators. Math. Program. 111(1), 173–199 (2008)

    MathSciNet  MATH  Google Scholar 

  17. Eckstein, J., Svaiter, B.F.: General projective splitting methods for sums of maximal monotone operators. SIAM J. Control Optim. 48(2), 787–811 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  18. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (Eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, chap. IX, pp. 299–340. North-Holland, Amsterdam (1983)

  19. Johnstone, P.R., Eckstein, J.: Projective splitting with forward steps. Math. Program. (2020). https://doi.org/10.1007/s10107-020-01565-3

    Article  MATH  Google Scholar 

  20. Johnstone, P.R., Eckstein, J.: Projective splitting with forward steps only requires continuity. Optim. Lett. 14(1), 229–247 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  21. Johnstone, P.R., Eckstein, J.: Convergence rates for projective splitting. SIAM J. Optim. 29(3), 1931–1957 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  22. Johnstone, P.R., Eckstein, J.: Single-forward-step projective splitting: exploiting cocoercivity. arXiv preprint arXiv:1902.09025 (2019)

  23. Korpelevich, G.: Extragradient method for finding saddle points and other problems. Matekon 13(4), 35–49 (1977)

    MathSciNet  Google Scholar 

  24. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  25. Ma, X.J., Wang, Z., Ryan, P.D., Isakoff, S.J., Barmettler, A., Fuller, A., Muir, B., Mohapatra, G., Salunga, R., Tuggle, J.T., et al.: A two-gene expression ratio predicts clinical outcome in breast cancer patients treated with tamoxifen. Cancer Cell 5(6), 607–616 (2004)

    Article  Google Scholar 

  26. Machado, M.P.: On the complexity of the projective splitting and Spingarn’s methods for the sum of two maximal monotone operators. J. Optim. Theory Appl. 178(1), 153–190 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  27. Machado, M.P.: Projective method of multipliers for linearly constrained convex minimization. Comput. Optim. Appl. 73(1), 237–273 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  28. Malitsky, Y., Pock, T.: A first-order primal-dual algorithm with linesearch. SIAM J. Optim. 28(1), 411–432 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  29. Malitsky, Y., Tam, M.K.: A forward-backward splitting method for monotone inclusions without cocoercivity. SIAM J. Optim. 30(2), 1451–1472 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  30. Michelot, C.: A finite algorithm for finding the projection of a point onto the canonical simplex of \(\mathbb{R}^n\). J. Optim. Theory Appl. 50(1), 195–200 (1986)

    MathSciNet  MATH  Google Scholar 

  31. Pedregosa, F., Gidel, G.: Adaptive three-operator splitting. In: Proceedings of the 35th International Conference on Machine Learning (ICML-18), pp. 4085–4094 (2018)

  32. Pesquet, J.C., Repetti, A.: A class of randomized primal-dual algorithms for distributed optimization. J. Nonlinear Convex Anal. 16(12), 2453–2490 (2015)

    MathSciNet  MATH  Google Scholar 

  33. Polyak, B.T.: Introduction to Optimization. Optimization Software Inc., Publications Division, New York (1987)

    MATH  Google Scholar 

  34. Simon, N., Friedman, J., Hastie, T., Tibshirani, R.: A sparse-group lasso. J. Comput. Graph. Stat. 22(2), 231–245 (2013)

    Article  MathSciNet  Google Scholar 

  35. Subramanian, A., Tamayo, P., Mootha, V.K., Mukherjee, S., Ebert, B.L., Gillette, M.A., Paulovich, A., Pomeroy, S.L., Golub, T.R., Lander, E.S., et al.: Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc. Natl. Acad. Sci. 102(43), 15545–15550 (2005)

    Article  Google Scholar 

  36. Tran-Dinh, Q., Vũ, B.C.: A new splitting method for solving composite monotone inclusions involving parallel-sum operators. Preprint arXiv:1505.07946 (2015)

  37. Tseng, P.: A modified forward–backward splitting method for maximal monotone mappings. SIAM J. Control Optim. 38(2), 431–446 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  38. Vũ, B.C.: A variable metric extension of the forward–backward–forward algorithm for monotone operators. Numer. Funct. Anal. Optim. 34(9), 1050–1065 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  39. Wang, H., Lu, Y., Zhai, C.: Latent aspect rating analysis on review text data: a rating regression approach. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 783–792 (2010)

  40. Yan, X., Bien, J.: Rare Feature Selection in High Dimensions. J. Am. Stat. Assoc. (2020). https://doi.org/10.1080/01621459.2020.1796677

    Article  Google Scholar 

Download references

Acknowledgements

We thank Xiaohan Yan and Jacob Bien for kindly sharing their data for the TripAdvisor reviews problem in Sect. 6.3. This research was supported by the National Science Foundation Grant CCF-1617617.

Author information

Authors and Affiliations

  1. Computational Science Initiative, Brookhaven National Laboratory, Upton, NY, USA

    Patrick R. Johnstone

  2. Department of Management Sciences and Information Systems, Rutgers Business School Newark and New Brunswick, Rutgers University, Piscataway, NJ, USA

    Jonathan Eckstein

Authors
  1. Patrick R. Johnstone
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Eckstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick R. Johnstone.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Johnstone, P.R., Eckstein, J. Single-forward-step projective splitting: exploiting cocoercivity. Comput Optim Appl 78, 125–166 (2021). https://doi.org/10.1007/s10589-020-00238-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-020-00238-3

Keywords

Search

Navigation