Abstract
In this paper, we combine the positive aspects of the gradient sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization problems. The main aim of the proposed method is to take advantage of both GS and bundle methods, meanwhile avoiding their drawbacks. At each iteration of this method, to find an efficient descent direction, the GS technique is utilized for constructing a local polyhedral model for the objective function. If necessary, via an iterative improvement process, this initial polyhedral model is improved by some techniques inspired by the bundle and GS methods. The convergence of the method is studied, which reveals that the global convergence property of our method is independent of the number of gradient evaluations required to establish and improve the initial polyhedral models. Thus, the presented method needs much fewer gradient evaluations in comparison to the original GS method. Furthermore, by means of numerical simulations, we show that the presented method provides promising results in comparison with GS methods, especially for large scale problems. Moreover, in contrast with some bundle methods, our method is not very sensitive to the accuracy of supplied gradients.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ackooij, W., Frangioni, A.: Incremental bundle methods using upper models. SIAM J. Optim. 28(1), 379–410 (2018)
Astorino, A., Gaudioso, M.: Polyhedral separability through successive LP. J. Optim. Theory Appl. 112(2), 265–293 (2002)
Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization. Springer, Berlin (2014)
Bonnans, J., Gilbert, J., Lemaréchal, C., Sagastizábal, C.A.: Numerical Optimization. Theoretical and Practical Aspects. Springer, Berlin (2006)
Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York (2004)
Burke, J.V., Curtis, F.E., Lewis, A.S., Overton, M.L., Simões, L.E.A.: Gradient sampling methods for nonsmooth optimization. In: Bagirov, A.M., Gaudioso, M., Karmitsa, N., Mäkelä, M.M., Taheri, S. (eds.) Numerical Nonsmooth Optimization: State of the Art Algorithms, pp. 201–225. Springer, Cham (2020)
Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15(3), 751–779 (2005)
Curtis, F.E., Overton, M.L.: A sequential quadratic programming algorithm for nonconvex, nonsmooth constrained optimization. SIAM J. Optim. 22(2), 474–500 (2012)
Curtis, F.E., Que, X.: An adaptive gradient sampling algorithm for nonconvex nonsmooth optimization. Optim. Methods Softw. 28(6), 1302–1324 (2013)
Evans, L.C., Gariepy, R.F.: Measure Theory and Fine Properties of Functions, Revised edn. CRC Press, Boca Raton (1992)
Haarala, M., Miettinen, K., Mäkelä, M.M.: New limited memory bundle method for large-scale nonsmooth optimization. Optim. Methods Softw. 19(6), 673–692 (2004)
Hare, W., Sagastizábal, C., Solodov, M.: A proximal bundle method for nonsmooth nonconvex functions with inexact information. Comput. Optim. Appl. 63(1), 1–28 (2016)
Helou, E.S., Santos, A.S., Simões, L.E.A.: On the differentiability check in gradient sampling methods. Optim. Methods Softw. 31(5), 983–1007 (2016)
Helou, E.S., Santos, A.S., Simões, L.E.A.: A fast gradient and function sampling method for finite-max functions. Comput. Optim. Appl. 71(3), 673–717 (2018)
Kiwiel, K.: An aggregate subgradient method for nonsmooth convex minimization. Math. Program. 27, 320–341 (1983)
Kiwiel, K.C.: Methods of Descent for Nondifferentiable Optimization. Springer, Berlin (1985)
Kiwiel, K.C.: Proximity control in bundle methods for convex nondifferentiable minimization. Math. Program. 46, 105–122 (1990)
Kiwiel, K.C.: Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J. Optim. 18(2), 379–388 (2007)
Kiwiel, K.C.: A nonderivative version of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J. Optim. 20(4), 1983–1994 (2010)
Lavor, C., Liberti, L., Maculan, N.: Molecular Distance Geometry Problem, 2nd edn. Springer, New York (2009)
Lukšan, L., Tcma, M., Siska, M., Vlček, J., Ramesova, N.: Ufo 2002. Interactive system for universal functional optimization. Tech. rep., Institute of Computer Science, Academy of Sciences of the Czech Republic (2002)
Lv, J., Pang, L., Xu, N., et al.: An infeasible bundle method for nonconvex constrained optimization with application to semi-infinite programming problems. Comput. Optim. Appl. 80(2), 397–427 (2019)
Mahdavi-Amiri, N., Shaeiri, M.: A conjugate gradient sampling method for nonsmooth optimization. 4OR 18, 73–90 (2019)
Mäkelä, M.M., Neittaanmäki, P.: Nonsmooth Optimization: Analysis and Algorithms with Applications to optimal control. World Scientific, Singapore (1992)
Nikolova, M.: Minimizers of cost-functions involving nonsmooth data-fidelity terms. Application to the processing of outliers. SIAM J. Numer. Anal. 40(3), 965–994 (2002)
Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. 148(1–2), 241–277 (2014)
Oliveira, W., Solodov, M.: A doubly stabilized bundle method for nonsmooth convex optimization. Math. Program. 156(1), 125–159 (2016)
Outrata, J., Kocvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints. Springer, Dordrecht (1998)
Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J. Optim. 2(1), 105–122 (1992)
Shor, N.Z.: Minimization Methods for Non-differentiable Functions. Springer, Berlin (1985)
Skaaja, M.: Limited memory BFGS for nonsmooth optimization. Master’s thesis, New York University (2010)
Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Elsevier, Amsterdam (2005)
Acknowledgements
The authors would like to thank the two anonymous referees and coordinating editor for their valuable comments, which improved the quality of the paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Maleknia, M., Shamsi, M. A new method based on the proximal bundle idea and gradient sampling technique for minimizing nonsmooth convex functions. Comput Optim Appl 77, 379–409 (2020). https://doi.org/10.1007/s10589-020-00213-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-020-00213-y