Abstract
In this paper we consider convex feasibility problems where the feasible set is given as the intersection of a collection of closed convex sets. We assume that each set is specified algebraically as a convex inequality, where the associated convex function is general (possibly non-differentiable). For finding a point satisfying all the convex inequalities we design and analyze random projection algorithms using special subgradient iterations and extrapolated stepsizes. Moreover, the iterate updates are performed based on parallel random observations of several constraint components. For these minibatch stochastic subgradient-based projection methods we prove sublinear convergence results and, under some linear regularity condition for the functional constraints, we prove linear convergence rates. We also derive sufficient conditions under which these rates depend explicitly on the minibatch size. To the best of our knowledge, this work is the first deriving conditions that show theoretically when minibatch stochastic subgradient-based projection updates have a better complexity than their single-sample variants when parallel computing is used to implement the minibatch. Numerical results also show a better performance of our minibatch scheme over its non-minibatch counterpart.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alamo, T., Tempo, R., Camacho, E.F.: Randomized strategies for probabilistic solutions of uncertain feasibility and optimization problems. IEEE Trans. Autom. Control 54(11), 2545–2559 (2009)
Bauschke, H.H.: Projection algorithms: Results and open problems. In: Butnariu, D., Censor, Y., Reich, S. (eds.) Inherently Parallel Algorithms in Feasibility and Optimization and their Applications, pp. 11–22. Elsevier, Amsterdam (2001)
Bauschke, H.H., Combettes, P.L., Luke, D.R.: Hybrid projection-reflection method for phase retrieval. J. Opt. Soc. Am. 20(6), 1025–1034 (2003)
Boyd, S., El Ghaoui, L., Feron, E., Balakrishnan, V.: Linear Matrix Inequalities in Systems and Control Theory. SIAM, Philadelphia (1994)
Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. Zh. Vychisl. Mat. Mat. Fiz. 7, 620–631 (1967)
Burke, J.V., Ferris, M.C.: Weak sharp minima in mathematical programming. SIAM J. Control Optim. 31(6), 1340–1359 (1993)
Blatt, D., Hero, A.: Energy based sensor network source localization via projection onto convex sets. IEEE Trans. Signal Process. 54(9), 3614–3619 (2006)
Calafiore, G., Polyak, B.T.: Stochastic algorithms for exact and approximate feasibility of robust LMI’s. IEEE Trans. Autom. Control 40(11), 1755–1759 (2001)
Combettes, P.L.: Convex set theoretic image recovery by extrapolated iterations of parallel subgradient projections. IEEE Trans. Image Process. 6(4), 493–506 (1997)
Combettes, P.L.: Hilbertian convex feasibility problem: convergence of projection methods. Appl. Math. Optim. 35, 311–330 (1997)
Deutsch, F.: Rate of convergence of the method of alternating projections. In: Brosowski, B., Deutsch, F. (eds.) Parametric Optimization and Approximation, pp. 96–107. Birkhauser, Basel (1983)
Deutsch, F., Hundal, H.: The rate of convergence for the cyclic projections algorithm I: Angles between convex sets. J. Approx. Theory 142, 36–55 (2006)
Facchinei, F., Pang, J.-S.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, Berlin (2003)
Fercoq, O., Alacaoglu, A., Necoara, I., Cevher, V.: Almost surely constrained convex optimization. Presented at the (2019)
Gubin, L.G., Polyak, B.T., Raik, E.V.: The method of projections for finding the common point of convex sets. Comput. Math. Math. Phys. 7(6), 1211–1228 (1967)
Guler, O., Hoffman, A., Rothblum, U.: Approximations to solutions to systems of linear inequalities. DIMACS technical report, DIMACS Center for Discrete Mathematics and Theoretical Computer Science (1992)
Halperin, I.: The product of projection operators. Acta Sci. Math. 23, 96–99 (1962)
Kundu, A., Bach, F., Bhattacharrya, C.: Convex optimization over intersection of simple sets: improved convergence rate guarantees via an exact penalty approach. In: International Conference on Artificial Intelligence and Statistics, (2018)
Kaczmarz, S.: Angenaherte Auflosung von Systemen linearer Gleichungen. Bull. Acad. Sci. Pologne A35, 355–357 (1937)
Lewis, A., Pang, J.: Error bounds for convex inequality systems. In: Crouzeix, J.P., Martinez-Legaz, J.E., Volle, M. (eds.) Generalized Convexity, Generalized Monotonicity: Recent Results. Nonconvex Optimization and Its Applications, pp. 75–110. Springer, Berlin (1998)
Liew, A., Yan, H., Law, N.: POCS-based blocking artifacts suppression using a smoothness constraint set with explicit region modeling. IEEE Trans. Circ. Syst. Video Tech. 15, 795–800 (2005)
Lacoste-Julien, S., Schmidt, M., Bach, F.: A simpler approach to obtaining an \(O(1/t)\) convergence rate for the projected stochastic subgradient method. CoRR, abs/1212.2002, (2012)
Motzkin, T.S., Shoenberg, I.: The relaxation method for linear inequalities. Can. J. Math. 6, 393–404 (1954)
Necoara, I., Richtarik, P., Patrascu, A.: Randomized projection methods for convex feasibility problems. SIAM J. Optim. 29(4), 2814–2852 (2019)
Necoara, I.: Faster randomized block Kaczmarz algorithms. SIAM J. Matrix Anal. Appl. 40(4), 1425–1452 (2019)
Nedic, A.: Random projection algorithms for convex set intersection problems. In: Proceedings of IEEE Conference on Decision and Control, pp. 7655–7660, (2010)
Nedic, A.: Random algorithms for convex minimization problems. Math. Progr. 129(2), 225–273 (2011)
Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Nesterov, Yu., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia (1994)
Polyak, B.T.: Minimization of unsmooth functionals. Comput. Math. Math. Phys. 9, 14–29 (1969)
Polyak, B.T.: Introduction to Optimization. Optimization Software, New York (1987)
Polyak, B.T.: Random algorithms for solving convex inequalities. In: Butnariu, D., Censor, Y., Reich, S. (eds.) Inherently Parallel Algorithms in Feasibility and Optimization and their Applications, pp. 409–422. Elsevier, Amsterdam (2001)
Patrascu, A., Necoara, I.: Nonasymptotic convergence of stochastic proximal point algorithms for constrained convex optimization. J. Mach. Learn. Res. 18(198), 1–42 (2018)
Peng, X., Li, L., Wang, F.: Accelerating minibatch stochastic gradient descent using typicality sampling. IEEE Trans. Neural Networks Learn. Syst. (2019). https://doi.org/10.1109/tnnls.2019.2957003
Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Stark, H., Yang, Y.: Vector space projections: A numerical approach to signal and image processing. Wiley-Interscience, Neural Nets and Optics (1998)
Acknowledgements
The research leading to these results has received funding from the NO Grants 2014–2021, under project ELO-Hyp, Contract No. 24/2020.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Necoara, I., Nedić, A. Minibatch stochastic subgradient-based projection algorithms for feasibility problems with convex inequalities. Comput Optim Appl 80, 121–152 (2021). https://doi.org/10.1007/s10589-021-00294-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-021-00294-3