Skip to main content
SpringerLink
Log in

Inexact stochastic subgradient projection method for stochastic equilibrium problems with nonmonotone bifunctions: application to expected risk minimization in machine learning

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

This paper discusses a stochastic equilibrium problem for which the function is in the form of the expectation of nonmonotone bifunctions and the constraint set is closed and convex. This problem includes various applications such as stochastic variational inequalities, stochastic Nash equilibrium problems, and nonconvex stochastic optimization problems. For solving this stochastic equilibrium problem, we propose an inexact stochastic subgradient projection method. The proposed method sets a random realization of the bifunction and then updates its approximation by using both its stochastic subgradient and the projection onto the constraint set. The main contribution of this paper is to present a convergence analysis showing that, under certain assumptions, any accumulation point of the sequence generated by the proposed method using a constant step size almost surely belongs to the solution set of the stochastic equilibrium problem. A convergence rate analysis of the method is also provided to illustrate the method’s efficiency. Another contribution of this paper is to show that a machine learning algorithm based on the proposed method achieves the expected risk minimization for a class of least absolute selection and shrinkage operator (lasso) problems in statistical learning with sparsity. Numerical comparisons of the proposed machine learning algorithm with existing machine learning algorithms for the expected risk minimization using LIBSVM datasets demonstrate the effectiveness and superior classification accuracy of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code availability

Not applicable.

Notes

  1. The existence of a solution to Problem 2.1 is guaranteed under the assumptions in Theorem 3.1 (see the proof of Theorem 3.1 for details).

  2. In this paper, we sometimes write \(\theta _\xi (\cdot )\) for a functional \(\theta (\cdot ;\xi ) :{\mathbb {R}}^N \rightarrow {\mathbb {R}}\) \((\xi \in { {\Xi }})\).

  3. The sequence \((v_n)_{n\in {\mathbb {N}}} \subset {\mathbb {R}}^N\) is said to be almost surely bounded if \(\sup \{ \Vert v_n\Vert :n\in {\mathbb {N}} \} < +\infty \) holds almost surely [10, (2.1.4)].

  4. This condition holds when C is bounded and \(F((\cdot ,\cdot );\xi )\) is pseudomonotone (see the proof of Theorem 3.1(ii)) or when \(F((\cdot ,\cdot );\xi )\) is defined by (6) and the solution set of the Minty variational inequality (19) is nonempty (see (20)).

  5. When s is sufficiently small, \({\tilde{r}}(w) := \sum _{k=1}^K \omega _k \min \{ \Vert w_{g_k}\Vert , s \Vert w_{g_k}\Vert + (1-s)c \} \approx r(w) := \sum _{k=1}^K \omega _k \min \{ \Vert w_{g_k}\Vert , c \}\) in the sense of the norm of \({\mathbb {R}}\).

  6. The function r defined by (34) is quasiconvex [62, Theorems 4.1 and 4.3] rather than pseudoconvex. Accordingly, we modify \((1/M)((1/2)l_i(w) + r(w))\) with \(\theta _i\) so that F defined by (37) can satisfy the strict pseudomonotonicity condition that is needed to guarantee the almost sure convergence of Algorithm 1 to the solution to Problem 2.1 with F defined by (37) (see Theorem 3.1(iii)).

References

  1. Androulakis, I.P., Maranasand, C.D., Flouda, C.A.: \(\alpha \)BB: a global optimization method for general constrained nonconvex problems. J. Global Optim. 7, 337–363 (1995)

    Article  MathSciNet  Google Scholar 

  2. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Lojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)

    Book  MATH  Google Scholar 

  5. Bauschke, H.H., Goebel, R., Lucet, Y., Wang, X.: The proximal average: basic theory. SIAM J. Optim. 19, 766–785 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bento, G.C., Cruz Neto, J.X., Lopes, J.O., Soares Jr., P.A., Soubeyran, A.: Generalized proximal distances for bilevel equilibrium problems. SIAM J. Optim. 26, 810–830 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bertsekas, D.P., Nedić, A., Ozdaglar, A.E.: Convex Analysis and Optimization. Athena Scientific, Belmont (2003)

    MATH  Google Scholar 

  8. Bianchi, M., Schaible, S.: Generalized monotone bifunctions and equilibrium problems. J. Math. Anal. Appl. 90, 31–43 (1996)

    MathSciNet  MATH  Google Scholar 

  9. Blum, E., Oettli, W.: From optimization and variational inequalities to equilibrium problems. The Mathematics Student 63, 123–145 (1994)

    MathSciNet  MATH  Google Scholar 

  10. Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  11. Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples, 2nd edn. Springer, New York (2000)

    Book  MATH  Google Scholar 

  12. Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 27:1–27:27 https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ (2011)

  13. Chen, X., Pong, T.K., Wets, R.J.B.: Two-stage stochastic variational inequalities: an ERM-solution procedure. Math. Program. 165, 71–111 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  14. Chen, X., Sun, H., Wets, R.J.B.: Regularized mathematical programs with stochastic equilibrium constraints: estimating structural demand models. SIAM J. Optim. 25, 53–75 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  15. Chen, Y., Lan, G., Ouyang, Y.: Accelerated schemes for a class of variational inequalities. Math. Program. 165, 113–149 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Cheung, Y., Lou, J.: Proximal average approximated incremental gradient descent for composite penalty regularized empirical risk minimization. Mach. Learn. 106, 595–622 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  17. Combettes, P.L., Hirstoaga, S.A.: Equilibrium programming in Hilbert spaces. J. Nonlinear Convex Anal. 6, 117–136 (2005)

    MathSciNet  MATH  Google Scholar 

  18. Crespi, G.P., Guerraggio, A., Rocca, M.: Minty variational inequality and optimization: scalar and vector case. In: Eberhard, A., Hadjisavvas, N., Luc, D.T. (eds.) Generalized Convexity, Generalized Monotonicity and Applications. Nonconvex Optimization and Its Applications, vol. 77. Springer, Boston (2005)

    MATH  Google Scholar 

  19. Crespi, G.P., Rocca, M.: Minty variational inequalities and monotone trajectories of differential inclusions. J. Inequal. Pure Appl. Math. 5, 48 (2004)

    MathSciNet  MATH  Google Scholar 

  20. Facchinei, F., Kanzow, C.: Penalty methods for the solution of generalized Nash equilibrium problems. SIAM J. Optim. 20, 2228–2253 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  21. Facchinei, F., Pang, J.S.: Finite-Dimensional Variational Inequalities and Complementarity Problems I. Springer, New York (2003)

    MATH  Google Scholar 

  22. Facchinei, F., Pang, J.S.: Finite-Dimensional Variational Inequalities and Complementarity Problems II. Springer, New York (2003)

    MATH  Google Scholar 

  23. Fan, L., Liu, S., Gao, S.: Generalized monotonicity and convexity of non-differentiable functions. J. Math. Anal. Appl. 279, 276–289 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  24. Floudas, C.A., Visweswaran, V.: A global optimization algorithm (GOP) for certain classes of nonconvex NLPs: I. Theory. Computers and Chemical Engineering 14, 1397–1417 (1990)

    Article  Google Scholar 

  25. Fukushima, M., Pang, J.S.: Some feasibility issues in mathematical programs with equilibrium constraints. SIAM J. Optim. 8, 673–681 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  26. Gao, D.Y.: Canonical duality theory and solutions to constrained nonconvex quadratic programming. J. Global Optim. 29, 377–399 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  27. Ghadimi, S., Lan, G.: Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: a generic algorithmic framework. SIAM J. Optim. 22, 1469–1492 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  28. Ghadimi, S., Lan, G.: Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization II: shrinking procedures and optimal algorithms. SIAM J. Optim. 23, 2061–2089 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  29. Goebel, K., Kirk, W.A.: Topics in Metric Fixed Point Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press, Cambridge (1990)

    Book  MATH  Google Scholar 

  30. Goebel, K., Reich, S.: Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings. Dekker, New York (1984)

    MATH  Google Scholar 

  31. Hastie, T., Tibshirani, R., Wainwright, M.: Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman and Hall/CRC, Boca Raton (2015)

    Book  MATH  Google Scholar 

  32. Iiduka, H.: Fixed point optimization algorithm and its application to power control in CDMA data networks. Math. Program. 133, 227–242 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  33. Iiduka, H.: Iterative algorithm for triple-hierarchical constrained nonconvex optimization problem and its application to network bandwidth allocation. SIAM J. Optim. 22, 862–878 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  34. Iiduka, H.: Stochastic fixed point optimization algorithm for classifier ensemble. IEEE Trans. Cybern. 50, 4370–4380 (2020)

    Article  Google Scholar 

  35. Iusem, A.N., Jofré, A., Oliveira, R.I., Thompson, P.: Extragradient method with variance reduction for stochastic variational inequalities. SIAM J. Optim. 27, 686–724 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  36. Iusem, A.N., Sosa, W.: Iterative algorithms for equilibrium problems. Optimization 52, 301–316 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  37. Iusem, A.N., Sosa, W.: New existence results for equilibrium problems. Nonlinear Anal. Theory Methods Appl. 52, 621–635 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  38. Karamardian, S.: Complementarity problems over cones with monotone and pseudomonotone maps. J. Optim. Theory Appl. 18, 445–454 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  39. Kinderlehrer, D., Stampacchia, G.: An Introduction to Variational Inequalities and Their Applications. Classics Appl. Math. 31. SIAM, Philadelphia (2000)

  40. King, A., Rockafellar, R.: Asymptotic theory for solutions in statistical estimation and stochastic programming. Math. Oper. Res. 18, 148–162 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  41. Kraft, D.: A software package for sequential quadratic programming. Tech. rep., DFVLR-FB 88-28, DLR German Aerospace Center–Institute for Flight Mechanics, Koln, Germany (1988)

  42. Lamm, M., Lu, S., Budhiraja, A.: Individual confidence intervals for solutions to expected value formulations of stochastic variational inequalities. Math. Program. 165, 151–196 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  43. Liu, Y., Xu, H.: Entropic approximation for mathematical programs with robust equilibrium constraints. SIAM J. Optim. 24, 933–958 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  44. Lu, Z., Li, X.: Sparse recovery via partial regularization: models, theory, and algorithms. Math. Oper. Res. 43, 1290–1316 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  45. Luo, Z.Q., Pang, J.S., Ralph, D.: Mathematical Programs with Equilibrium Constraints. Cambridge University Press, Cambridge (1996)

    Book  MATH  Google Scholar 

  46. Nash, J.F.: Equilibrium points in \(n\)-person games. Proc. Nat. Acad. Sci. 36, 48–49 (1950)

    Article  MathSciNet  MATH  Google Scholar 

  47. Nash, J.F.: Non-cooperative games. Ann. Math. 54, 286–295 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  48. Nedić, A., Lee, S.: On stochastic subgradient mirror-descent algorithm with weighted averaging. SIAM J. Optim. 24, 84–107 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  49. Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19, 1574–1609 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  50. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)

    Google Scholar 

  51. Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, Amsterdam (1970)

    MATH  Google Scholar 

  52. Pang, J.S., Sen, S.E., Shanbhag, U.V.: Two-stage non-cooperative games with risk-averse players. Math. Program. 165, 235–290 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  53. Ravat, U., Shanbhag, U.V.: On the existence of solutions to stochastic quasi-variational inequality and complementarity problems. Math. Program. 165, 291–330 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  54. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  55. Rockafellar, R.T., Wets, R.J.B.: Variational Analysis. Springer, Berlin (2010)

    MATH  Google Scholar 

  56. Saigal, R.: Extension of the generalized complementarity problem. Math. Oper. Res. 1, 260–266 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  57. Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: Primal estimated sub-gradient solver for SVM. Math. Program. 127, 3–30 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  58. Shanbhag, U.V., Pang, J.S., Sen, S.: Inexact best-response schemes for stochastic Nash games: Linear convergence and iteration complexity analysis. In: 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 3591–3596 (2016)

  59. Shapiro, A., Dentcheva, D., Ruszczyński, A.: Lectures on Stochastic Programming: Modeling and Theory. MOS-SIAM Series on Optimization, 2nd edn. SIAM, Philadelphia (2014)

    MATH  Google Scholar 

  60. Sundhar Ram, S., Nedić, A., Veeravalli, V.V.: Incremental stochastic subgradient algorithms for convex optimization. SIAM J. Optim. 20, 691–717 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  61. Sundhar Ram, S., Nedić, A., Veeravalli, V.V.: Distributed stochastic subgradient projection algorithms for convex optimization. J. Optim. Theory Appl. 147, 516–545 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  62. Suzuki, S.: Quasiconvexity of sum of quasiconvex functions. Linear Nonlinear Anal. 3, 287–295 (2017)

    MathSciNet  MATH  Google Scholar 

  63. Tada, A., Takahashi, W.: Weak and strong convergence theorems for a nonexpansive mapping and an equilibrium problem. J. Optim. Theory Appl. 133, 359–370 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  64. Takahashi, W.: Nonlinear Functional Analysis. Yokohama Publishers, Yokohama (2000)

    MATH  Google Scholar 

  65. Vapnik, V.N.: Statistical Learning Theory. Wiley, Canada (1998)

    MATH  Google Scholar 

  66. Wang, M., Bertsekas, D.P.: Stochastic first-order methods with random constraint projection. SIAM J. Optim. 26, 681–717 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  67. Xu, H., Ye, J.J.: Necessary optimality conditions for two-stage stochastic programs with equilibrium constraints. SIAM J. Optim. 20, 1685–1715 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  68. Xu, Y., Lin, Q., Yang, T.: Accelerated stochastic subgradient methods under local error bound condition. arXiv, https://arxiv.org/abs/1607.01027

  69. Yao, J.C.: Multi-valued variational inequalities with \({K}\)-pseudomonotone operators. J. Optim. Theory Appl. 83, 391–403 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  70. Yousefian, F., Nedić, A., Shanbhag, U.V.: On smoothing, regularization, and averaging in stochastic approximation methods for stochastic variational inequality problems. Math. Program. 165, 391–431 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  71. Zhang, T.: Analysis of multi-stage convex relaxation for sparse regularization. J. Mach. Learn. Res. 11, 1081–1107 (2010)

    MathSciNet  MATH  Google Scholar 

  72. Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. Chapman and Hall, Boca Raton (2012)

    Book  Google Scholar 

Download references

Acknowledgements

I am sincerely grateful to Editor-in-Chief Sergiy Butenko and the two anonymous reviewers for helping me improve the original manuscript. I also thank Kazuhiro Hishinuma for his input on the numerical examples.

Author information

Authors and Affiliations

  1. Department of Computer Science, Meiji University, 1-1-1 Higashimita, Tama-ku, Kawasaki-shi, Kanagawa, 214-8571, Japan

    Hideaki Iiduka

Authors
  1. Hideaki Iiduka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hideaki Iiduka.

Ethics declarations

Conflict of interest

The author declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by JSPS KAKENHI Grant Number JP18K11184.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iiduka, H. Inexact stochastic subgradient projection method for stochastic equilibrium problems with nonmonotone bifunctions: application to expected risk minimization in machine learning. J Glob Optim 80, 479–505 (2021). https://doi.org/10.1007/s10898-020-00980-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-020-00980-2

Keywords

Mathematics Subject Classification

Search

Navigation