Skip to main content
SpringerLink
Log in

A primal perspective for indefinite kernel SVM problem

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Indefinite kernel support vector machine (IKSVM) has recently attracted increasing attentions in machine learning. Since IKSVM essentially is a non-convex problem, existing algorithms either change the spectrum of indefinite kernel directly but risking losing some valuable information or solve the dual form of IKSVM whereas suffering from a dual gap problem. In this paper, we propose a primal perspective for solving the problem. That is, we directly focus on the primal form of IKSVM and present a novel algorithm termed as IKSVM-DC for binary and multi-class classification. Concretely, according to the characteristics of the spectrum for the indefinite kernel matrix, IKSVM-DC decomposes the primal function into the subtraction of two convex functions as a difference of convex functions (DC) programming. To accelerate convergence rate, IKSVM-DC combines the classical DC algorithm with a line search step along the descent direction at each iteration. Furthermore, we construct a multi-class IKSVM model which can classify multiple classes in a unified form. A theoretical analysis is then presented to validate that IKSVM-DC can converge to a local minimum. Finally, we conduct experiments on both binary and multi-class datasets and the experimental results show that IKSVM-DC is superior to other state-of-the-art IKSVM algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cortes C. Support-vector network. Machine Learning Journal, 1995, 20(3): 3–273

    Google Scholar 

  2. Cristianini N, Shawe-Taylor J. An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge: Cambridge University Press, 2000

    Book  MATH  Google Scholar 

  3. Li Y Y, Lu R Q. Locality preserving projection on SPD matrix Lie group: algorithm and analysis. Science China Information Sciences, 2018, 61(9): 092104

    Article  MathSciNet  Google Scholar 

  4. Ma L R, Song D D, Liao L J, Wang J. PSVM: a preference-enhanced SVM model using preference data for classification. Science China Information Sciences, 2017, 60(12): 122103

    Article  Google Scholar 

  5. Saigo H, Vert J-P, Ueda N, Akutsu T. Protein homology detection using string alignment kernels. Bioinformatics, 2004, 20(11): 11–1682

    Article  Google Scholar 

  6. Wang C, Song Y, Li H, Zhang M, Han J. Text classification with heterogeneous information network kernels. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016, 2130–2136

    Google Scholar 

  7. Vapnik V. The Nature of Statistical Learning Theory. Germany: Springer Science & Business Media, 2013

    MATH  Google Scholar 

  8. Suard F, Rakotomamonjy A, Bensrhair A. Kernel on bag of paths for measuring similarity of shapes. In: Proceedings of European Symposium on Artificial Neural Networks. 2007, 355–360

    Google Scholar 

  9. Chen Y, Garcia E, Gupta M, Rahimi A, Cazzanti L. Similarity-based classification: concepts and algorithms. Journal of Machine Learning Research, 2009, 10(3): 3–747

    MathSciNet  MATH  Google Scholar 

  10. Chen Y, Gupta M. Fusing similarities and kernels for classification, In: Proceedings of the 12th International Conference on IEEE Information Fusion. 2009, 474–481

    Google Scholar 

  11. Pkalska E, Haasdonk B. Kernel discriminant analysis for positive definite and indefinite kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(6): 6–1017

    Google Scholar 

  12. Sun H, Wu Q. Least square regression with indefinite kernels and coefficient regularization. Applied and Computational Harmonic Analysis, 2011, 30(1): 96–109

    Article  MathSciNet  MATH  Google Scholar 

  13. Ying Y, Campbell C, Girolami M. Analysis of svm with indefinite kernels. In: Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2009, 2205–2213

    Google Scholar 

  14. Loosli G, Canu S. Non positive SVM. Technical Report, 2010

    MATH  Google Scholar 

  15. Haasdonk B. Feature space interpretation of SVMs with indefinite kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(4): 4–482

    Article  Google Scholar 

  16. Pekalska E, Paclik P, Duin R P. A generalized kernel approach to dissimilarity-based classification. Journal of Machine Learning Research, 2001, 2(12): 12–175

    MATH  Google Scholar 

  17. Graepel T, Herbrich R, Bollmann-Sdorra P, Obermayer K. Classification on pairwise proximity data. In: Proceedings of the 13th Conference on Neural Information Processing Systems. 1999, 438–444

    Google Scholar 

  18. Roth V, Laub J, Kawanabe M, Buhmann J M. Optimal cluster preserving embedding of nonmetric proximity data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(12): 12–1540

    Article  Google Scholar 

  19. Luss R, d'Aspremont A. Support vector machine classification with indefinite kernels. In: Proceedings of the 21st Conference on Neural Information Processing Systems. 2008, 953–960

    Google Scholar 

  20. Chen J, Ye J. Training SVM with indefinite kernels. In: Proceedings of the 25th International Conference on Machine Learning. 2008, 136–143

    Google Scholar 

  21. Chen Y, Gupta M R, Recht B. Learning kernels from indefinite similarities. In: Proceedings of the 26th International Conference on Machine Learning. 2009, 145–152

    Google Scholar 

  22. Gu S, Guo Y. Learning SVM classifiers with indefinite kernels. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence. 2012, 942–948

    Google Scholar 

  23. Lin H T, Lin C J. A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods. Neural Computation, 2003, 3: 1–32

    Google Scholar 

  24. Akoa F B. Combining DC algorithms (DCAs) and decomposition techniques for the training of nonpositive-semidefinite kernels. IEEE Transactions on Neural Networks, 2008, 19(11): 11–1854

    Article  Google Scholar 

  25. Ong C S, Mary X, Canu S, Smola A J. Learning with non-positive kernels. In: Proceedings of the 21st International Conference on Machine Learning. 2004, 81

    Google Scholar 

  26. Loosli G, Canu S, Ong C S. Learning SVM in Krein spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(6): 6–1204

    Article  Google Scholar 

  27. Alabdulmohsin I M, Gao X, Zhang X. Support vector machines with indefinite kernels. In: Proceedings of the 6th Asian Conference on Machine Learning. 2014, 32–47

    Google Scholar 

  28. Kotsiantis S B, Zaharakis I, Pintelas P. Supervised machine learning: a review of classification techniques. Emerging Artificial Intelligence Applications in Computer Engineering, 2007, 160: 3–24

    Google Scholar 

  29. Friedman J. Another approach to polychotomous classification. Technical Report, Department of Statistics, Stanford University, 1996

    Google Scholar 

  30. Krebel U. Pairwise classification and support vector machines. Advances in Kernel Methods: Support Vector Learning, 1999, 255–268

    Google Scholar 

  31. Bottou L, Cortes C, Denker J S, Drucker H, Guyon Z, Jackel L D, Le Cun Y, Muller U A, Sackinger E, Simard P, Vapnik U. Comparison of classifier methods: a case study in handwritten digit recognition. In: Proceedings of the 12th IAPR International Conference on Pattern Recognition. 1994, 77–82

    Google Scholar 

  32. Dietterich T G, Bakiri G. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 1995, 2: 263–286

    Article  MATH  Google Scholar 

  33. Allwein E L, Schapire R E, Singer Y. Reducing multiclass to binary: a unifying approach for margin classifiers. Journal of Machine Learning Research, 2000, 1(12): 12–113

    MATH  Google Scholar 

  34. Piatt J C, Cristianini N, Shawe-Taylor J. Large margin dags for multi-class classification. In: Proceedings of the 12th International Conference on Neural Information Processing Systems. 1999, 547–553

    Google Scholar 

  35. Scholkopf B, Smola A J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Massackusetts: MIT Press, 2001

    Google Scholar 

  36. Tao P D, An L T H. Convex analysis approach to DC programming: theory, algorithms and applications. Acta Mathematica Vietnamica, 1997, 22(1): 1–289

    Google Scholar 

  37. Dinh T P, Le Thi H A. Recent advances in DC programming and DCA. In: Nguyen N T, Le-Thi H A, eds. Transactions on Computational Intelligence XIII. Springer, Berlin, Heidelbery, 2014, 1–37

    Google Scholar 

  38. Piot B, Geist M, Pietquin O. Difference of convex functions programming for reinforcement learning. In: Proceedings of the 27th Conference on Neural Information Processing Systems. 2014, 2519–2527

    Google Scholar 

  39. Xu H M, Xue H, Chen X H, Wang Y Y. Solving indefinite kernel support vector machine with difference of convex functions programming. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017, 2782–2788

    Google Scholar 

  40. Artacho F J A, Fleming R M, Vuong P T. Accelerating the DC algo-rithmfor smooth functions. Mathematical Programming, 2018, 169(1): 1–95

    Article  MathSciNet  Google Scholar 

  41. Nesterov Y. Introductory Lectures on Convex Optimization: A Basic Course. Germany: Springer Science & Business Media, 2013

    MATH  Google Scholar 

  42. Wang Z, Xue X. Multi-class support vector machine. In: Ma Y, Guo G, eds. Support Vector Machines Applications. Springer, Cham, 2014, 23–48

    Chapter  Google Scholar 

  43. Vapnik V N. Statistical Learning Theory. Wiley New York: John Wiley & Sons, Inc., 1998

    MATH  Google Scholar 

  44. Weston J, Watkins C. Multi-class support vector machines. Technical Report, Department of Computer Science, Royal Holloway, University of London, 1998

    Google Scholar 

  45. Crammer K, Singer Y. On the leamability and design of output codes for multi-class problems. Machine Learning, 2002, 47(2): 2–201

    Google Scholar 

  46. Blake C, Merz C J. UCI repository of machine learning databases, 1998

    Google Scholar 

  47. Ratsch G, Onoda T, Muller K R. Soft margins for AdaBoost. Machine Learning, 2001, 42(3): 3–287

    Article  MATH  Google Scholar 

  48. Duin R P, Pekalska E. The dissimilarity representation for pattern recognition: a tutorial. Technical Report, 2009

    Google Scholar 

  49. Mosek A. The MOSEK optimization software. Google Website, 2010

    Google Scholar 

  50. Wu G, Chang E Y, Zhang Z. An analysis of transformation on non-positive semidefinite similarity matrix for kernel machines. In: Proceedings of the 22nd International Conference on Machine Learning. 2005, 1245–1256

    Google Scholar 

  51. Chang C C, Lin C J. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2011, 2(3): 27

    Article  Google Scholar 

  52. Zhang M L, Zhou Z H. Exploiting unlabeled data to enhance ensemble diversity. Data Mining and Knowledge Discovery, 2013, 26(1): 98–129

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Key R&D Program of China (2017YFB1002801), the National Natural Science Foundations of China (Grant Nos. 61375057, 61876091 and 61403193). It was also supported by Collaborative Innovation Center of Wireless Communications Technology.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, 210096, China

    Hui Xue & Haiming Xu

  2. Key Laboratory of Computer Network and Information Integration (Southeast University), MOE, Nanjing, 210096, China

    Hui Xue & Haiming Xu

  3. College of Science, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Xiaohong Chen

  4. School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210046, China

    Yunyun Wang

Authors
  1. Hui Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haiming Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaohong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunyun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hui Xue.

Additional information

Hui Xue received the BSc degree in Mathematics from Nanjing Normal University, China in 2002. In 2005, she received the MSc degree in Mathematics from Nanjing University of Aeronautics & Astronautics (NUAA), China. And she also received the PhD degree in Computer Application Technology at NUAA, China in 2008. Since 2009, as an Associate Professor, she has been with the School of Computer Science and Engineering at Southeast University, China. Her research interests include pattern recognition and machine learning.

Harming Xu received the BS degree in School of Computer Science and Technology from China University of Mining and Techology, China in 2015. In 2018, he received the MSc degree in the School of Computer Science and Engineering at Southeast University, China. Now, he is a PhD candidate at the University of Adelaide, Australia. His research interests include pattern recognition, machine learning, and data mining.

Xiaohong Chen received the BSc degree in Mathematics from Qufu Normal University, China in 1998. In 2001, she received the MSc degree in Mathematics from Nanjing University of Aeronautics & Astronautics (NUAA), China. And she also received the PhD degree in Computer Application Technology at NUAA, China in 2011. She is an Associate Professor at the College of Science at NUAA, China. Her research interests include pattern recognition and machine learning.

Yunyun Wang received the BS degree in computer science and technology from An-hui Normal University, China in 2006, and the PhD degree in computer science and engineering from the Nanjing University of Aeronautics and Astronautics, China in 2012. She is currently an associate professor with the Department of Computer Science and Engineering, Nanjing University of Posts & Telecommunications, China. Her current research interests include pattern recognition and machine learning.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, H., Xu, H., Chen, X. et al. A primal perspective for indefinite kernel SVM problem. Front. Comput. Sci. 14, 349–363 (2020). https://doi.org/10.1007/s11704-018-8148-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-018-8148-z

Keywords

Search

Navigation