Skip to main content
SpringerLink
Log in

Kernel-based regression via a novel robust loss function and iteratively reweighted least squares

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Least squares kernel-based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottleneck. To address this issue, a generalized loss function, called \(\ell _s\)-loss, is proposed in this paper. With the support of novel loss function, two kernel-based regressors are constructed by replacing the \(\ell _2\)-loss in LS-SVR and ELM with the proposed \(\ell _s\)-loss for better noise robustness. Important properties of \(\ell _s\)-loss, including robustness, asymmetry and asymptotic approximation behaviors, are verified theoretically. Moreover, iteratively reweighted least squares are utilized to optimize and interpret the proposed methods from a weighted viewpoint. The convergence of the proposal is proved, and detailed analyses of robustness are given. Experiments on both artificial and benchmark datasets confirm the validity of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems, pp 1097–1105

  2. Lecun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551. https://doi.org/10.1162/neco.1989.1.4.541

    Article  Google Scholar 

  3. Audibert JY, Catoni O (2011) Robust linear least squares regression. Ann Stat 39(5):2766–2794. https://doi.org/10.1214/11-AOS918

    Article  MathSciNet  MATH  Google Scholar 

  4. Cheung YM, Zeng H (2009) Local kernel regression score for selecting features of high-dimensional data. IEEE Trans Knowl Data Eng 21(12):1798–1802. https://doi.org/10.1109/TKDE.2009.23

    Article  Google Scholar 

  5. Suykens JAK, Gestel TV, Brabanter JD, Moor BD, Vandewalle J (2002) Least squares support vector machines. Int J Circuit Theory Appl 27(6):605–615

    Article  Google Scholar 

  6. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501. https://doi.org/10.1016/j.neucom.2005.12.126

    Article  Google Scholar 

  7. Vapnik VN (2008) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  8. Bartlett P, Mendelson S (2006) Empirical minimization. Probab Theory Relat Field 135(3):311–334. https://doi.org/10.1007/s00440-005-0462-3

    Article  MathSciNet  MATH  Google Scholar 

  9. Fama F, MacBeth D, Jackel LD (1973) Risk, return, and equilibrium: empirical tests. J Polit Econ 81(3):607–636. https://doi.org/10.1086/260061

    Article  Google Scholar 

  10. Catoni O (2010) Challenging the empirical mean and empirical variance: a deviation study. Ann Inst Henri Poincare-Probab Stat 48(4):1148–1185. https://doi.org/10.1214/11-AIHP454

    Article  MathSciNet  MATH  Google Scholar 

  11. Kallummil S, Kalyani S (2019) Noise statistics oblivious GARD for robust regression with sparse outliers. IEEE Trans Signal Process 67(2):383–398. https://doi.org/10.1109/TSP.2018.2883025

    Article  MathSciNet  MATH  Google Scholar 

  12. Christmann A, Steinwart I (2007) Consistency and robustness of kernel based regression. Bernoulli 13(3):799–819. https://doi.org/10.3150/07-BEJ5102

    Article  MathSciNet  MATH  Google Scholar 

  13. Huang D, Cabral R, Torre FDL (2016) Robust regression. IEEE Trans Pattern Anal Mach Intell 38(2):363–375. https://doi.org/10.1109/TPAMI.2015.2448091

    Article  Google Scholar 

  14. Zhang L, Zhou ZH (2018) ł\(_1\)-regression with heavy-tailed distributions. In: Proceedings of Advances in Neural Information Processing Systems

  15. Yao Q, Tong H (2007) Asymmetric least squares regression estimation: a nonparametric approach. J Nonparametr Stat 6(4):273–292. https://doi.org/10.1080/10485259608832675

    Article  MathSciNet  MATH  Google Scholar 

  16. Chen C, Li Y, Yan C, Guo J, Liu G (2017) Least absolute deviation-based robust support vector regression. Knowledge Based Syst 131(1):183–194. https://doi.org/10.1016/j.knosys.2017.06.009

    Article  Google Scholar 

  17. Chen C, Yan C, Li Y (2015) A robust weighted least squares support vector regression based on least trimmed squares. Neurocomputing 168(30):941–946. https://doi.org/10.1016/j.neucom.2015.05.031

    Article  Google Scholar 

  18. Mangasarian OL, Musicant DR (2002) Robust linear and support vector regression. IEEE Trans Pattern Anal Mach Intell 22(9):950–955. https://doi.org/10.1109/34.877518

    Article  Google Scholar 

  19. Huber PJ (2014) Robust statistics. Springer, New York

    Google Scholar 

  20. Huber PJ (1964) Robust estimation of a location parameter. Ann Math Stat 35(1):73–101. https://doi.org/10.1214/aoms/1177703732

    Article  MathSciNet  MATH  Google Scholar 

  21. Christmann A, Steinwart I (2007) How svms can estimate quantiles and the median. In: Proceedings of Advances in Neural Information Processing Systems, pp 305–312

  22. Karal O (2017) Maximum likelihood optimal and robust support vector regression with lncosh loss function. Neural Netw 94(10):1–12. https://doi.org/10.1016/j.neunet.2017.06.008

    Article  MATH  Google Scholar 

  23. Ren Z, Yang Y (2018) Correntropy-based robust extreme learning machine for classification. Neurocomputing 313(11):74–84. https://doi.org/10.1016/j.neucom.2018.05.100

    Article  Google Scholar 

  24. Kai Z, Luo M (2015) Outlier-robust extreme learning machine for regression problems. Neurocomputing 151(3):1519–1527. https://doi.org/10.1016/j.neucom.2014.09.022

    Article  Google Scholar 

  25. Yang L, Dong H (2018) Support vector machine with truncated pinball loss and its application in pattern recognition. Chemom Intell Lab Syst 177(6):89–99. https://doi.org/10.1016/j.chemolab.2018.04.003

    Article  Google Scholar 

  26. Yang L, Dong H (2019) Robust support vector machine with generalized quantile loss for classification and regression. Appl Soft Comput 81(8):105483. https://doi.org/10.1016/j.asoc.2019.105483

    Article  Google Scholar 

  27. Holland MJ, Ikeda K (2017) Robust regression using biased objectives. Mach Learn 106(4):1–37. https://doi.org/10.1007/s10994-017-5653-5

    Article  MathSciNet  MATH  Google Scholar 

  28. Lugosi G, Mendelson S (2020) Risk minimization by median-of-means tournaments. J Eur Math Soc 22(3):925–965. https://doi.org/10.4171/JEMS/937

    Article  MathSciNet  MATH  Google Scholar 

  29. Catoni O (2009) High confidence estimates of the mean of heavy-tailed real random variables. arXiv:0909.5366

  30. Suykens JAK, Brabanter JD, Lukas L, Vandewalle J (2002) Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing 48(10):85–105. https://doi.org/10.1016/s0925-2312(01)00644-0

    Article  MATH  Google Scholar 

  31. Wang K, Zhong P (2014) Robust non-convex least squares loss function for regression with outliers. Knowl Based Syst 71:290–302. https://doi.org/10.1016/j.knosys.2014.08.003

    Article  Google Scholar 

  32. Zhao Y, Sun J (2010) Robust truncated support vector regression. Expert Syst Appl 37(7):5126–5133. https://doi.org/10.1016/j.eswa.2009.12.082

    Article  Google Scholar 

  33. Chen K, Lv Q, Lu Y, Dou Y (2016) Robust regularized extreme learning machine for regression using iteratively reweighted least squares. Neurocomputing 230(12):345–358. https://doi.org/10.1016/j.neucom.2016.12.029

    Article  Google Scholar 

  34. Dinh DP, Thi HAL, Akoa F (2008) Combining DCA (DC Algorithms) and interior point techniques for large-scale nonconvex quadratic programming. Optim Methods Softw 23(4):609–629. https://doi.org/10.1080/10556780802263990

    Article  MathSciNet  MATH  Google Scholar 

  35. Yang L, Qian Y (2016) A sparse logistic regression framework by difference of convex functions programming. Appl Intell 45(2):241–254. https://doi.org/10.1007/s10489-016-0758-2

    Article  Google Scholar 

  36. Yuille AL (2002) CCCP algorithms to minimize the Bethe and Kikuchi free energies: convergent alternatives to belief propagation. Neural Comput 14(7):1691–1722. https://doi.org/10.1162/08997660260028674

    Article  MATH  Google Scholar 

  37. Zhang Y, Sun Y, He R, Tan T (2013) Robust subspace clustering via half-quadratic minimization. In: Proceedings of IEEE International Conference on Computer Vision, pp 3096–3103

  38. He R, Zheng W, Tan T, Sun Z (2014) Half-quadratic-based iterative minimization for robust sparse representation. IEEE Trans Pattern Anal Mach Intell 36(2):261–275. https://doi.org/10.1109/TPAMI.2013.102

    Article  Google Scholar 

  39. Feng Y, Yang Y, Huang X, Mehrkanoon S, Suykens JAK (2016) Robust support vector machines for classification with nonconvex and smooth losses. Neural Comput 28(6):1217–1247. https://doi.org/10.1162/NECO_a_00837

    Article  MathSciNet  MATH  Google Scholar 

  40. Li C, Zhou S (2017) Sparse algorithm for robust LSSVM in primal space. Neurocomputing 275:2880–2891. https://doi.org/10.1016/j.neucom.2017.10.011

    Article  Google Scholar 

  41. Xu G, Hu B, Principe JC (2016) Robust C-loss kernel classifiers. IEEE Trans Neural Netw Learn Syst 29(3):510–522. https://doi.org/10.1109/TNNLS.2016.2637351

    Article  MathSciNet  Google Scholar 

  42. Lai MJ, Xu Y, Yin W (2013) Improved iteratively reweighted least squares for unconstrained smoothed łq minimization. SIAM J Numer Anal 51(2):927–957. https://doi.org/10.1137/110840364

    Article  MathSciNet  MATH  Google Scholar 

  43. Green PJ (1984) Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. J R Stat Soc Ser B-Stat Methodol 46(2):149–192. https://doi.org/10.1111/j.2517-6161.1984.tb01288.x

    Article  MathSciNet  MATH  Google Scholar 

  44. Debruyne M, Christmann A, Hubert M, Suykens JAK (2010) Robustness of reweighted least squares kernel based regression. J Multivar Anal 101:447–463. https://doi.org/10.1016/j.jmva.2009.09.007

    Article  MathSciNet  MATH  Google Scholar 

  45. Yi S, He Z, Cheung YM, Chen WS (2018) Unified sparse subspace learning via self-contained regression. IEEE Trans Circuits Syst Video Technol 28(10):2537–2550. https://doi.org/10.1109/TCSVT.2017.2721541

    Article  Google Scholar 

  46. Deng W, Zheng Q, Chen L (2009) Regularized extreme learning machine. In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining, pp 389–395 https://doi.org/10.1109/CIDM.2009.4938676

  47. Christmann A, Steinwart I (2008) Support vector machines. Springer, New York

    MATH  Google Scholar 

  48. Hampel FR, Ronchetti EM, Rousseeuw PJ, Stahel WA (1986) Robust statistics: the approach based on influence functions. Wiley, New York

    MATH  Google Scholar 

  49. Huang X, Shi L, Suykens JAK (2014) Support vector machine classifier with pinball loss. IEEE Trans Pattern Anal Mach Intell 36(5):984–997. https://doi.org/10.1109/TPAMI.2013.1786

    Article  Google Scholar 

  50. Dua D, Graff C (2019) UCI machine learning repository. (http://archive.ics.uci.edu/ml)

  51. Deng N, Tian Y, Zhang C (2012) Support vector machines: optimization based theory, algorithms and extensions. CRC Press, Boca Raton

    Book  Google Scholar 

  52. Crambes C, Gannoun A, Henchiri Y (2013) Support vector machine quantile regression approach for functional data: simulation and application studies. J Multivar Anal 121(11):50–68. https://doi.org/10.1016/j.jmva.2013.06.004

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work is supported by National Nature Science Foundation of China (Nos. 11471010, 11271367).

Author information

Authors and Affiliations

  1. College of Science, China Agricultural University, Beijing, 100083, China

    Hongwei Dong & Liming Yang

Authors
  1. Hongwei Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liming Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, H., Yang, L. Kernel-based regression via a novel robust loss function and iteratively reweighted least squares. Knowl Inf Syst 63, 1149–1172 (2021). https://doi.org/10.1007/s10115-021-01554-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-021-01554-8

Keywords

Search

Navigation