Skip to main content
SpringerLink
Log in

A novel binary many-objective feature selection algorithm for multi-label data classification

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

For Multi-label classification, redundant and irrelevant features degrade the performance of classification. To select the best features based on several conflicting objectives, feature selection can be modeled as a large-scale optimization problem. However, most existing multi-objective feature selection methods select the features based on minimizing two well-known objectives, the number of features and classification error, additional objectives can be considered to improve the classification performance. In this study, for the first time, a many-objective optimization method is proposed to select the efficient features for multi-label classification based on not only two mentioned objectives, but also maximizing the correlation between features and labels and minimizing the computational complexity of features. Maximizing the correlation could lead to increasing the accuracy of classification. On the other hand, selecting less complex features decreases the computational complexity of feature extraction phase. The most important aim of this paper is to tackle the multi-label feature selection based on the number of features, classification error, correlation between features and labels, and computational complexity of features, simultaneously. The conducted many-objective feature selection problem is solved using a proposed binary version of NSGA-III algorithm. The binary operator improves the exploration power of optimizer to search the large-scale space. In order to evaluate the proposed algorithm (called binary NSGA-III), a benchmarking experiments is conducted on eight multi-label datasets in terms of several multi-objective assessment metrics, including Hypervolume indicator, Pure Diversity, and Set-coverage. Experimental results show significant improvements for proposed method in comparison with other algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Chen G, Ye D, Xing Z, Chen J, Cambria E (2017) Ensemble application of convolutional and recurrent neural networks for multi-label text categorization. In: International joint conference on neural networks (IJCNN), IEEE, pp 2377–2383

  2. Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W (2016) CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2285–2294

  3. Li Z, Liao B, Li Y, Liu W, Chen M, Cai L (2018) Gene function prediction based on combining gene ontology hierarchy with multi-instance multi-label learning. RSC Adv 8(50):28503–28509

    Article  Google Scholar 

  4. Ueda N, Saito K (2003) Parametric mixture models for multi-labeled text. In: Advances in neural information processing systems, pp 737–744

  5. Boutell MR, Luo J, Shen X, Brown CM (2004) Learning multi-label scene classification. Pattern Recogn 37(9):1757–1771

    Article  Google Scholar 

  6. Li F, Miao D, Pedrycz W (2017) Granular multi-label feature selection based on mutual information. Pattern Recogn 67:410–423

    Article  Google Scholar 

  7. Lee J, Kim DW (2015) Mutual information-based multi-label feature selection using interaction information. Expert Syst Appl 42(4):2013–2025

    Article  Google Scholar 

  8. Xue B, Zhang M, Browne WN (2013) Particle swarm optimization for feature selection in classification: a multi-objective approach. IEEE Trans Cybern 43(6):1656–1671

    Article  Google Scholar 

  9. Wang CM, Huang YF (2009) Evolutionary-based feature selection approaches with new criteria for data mining: a case study of credit approval data. Expert Syst Appl 36(3):5900–5908

    Article  Google Scholar 

  10. Hagenau M, Liebmann M, Hedwig M, Neumann D (2012) Automated news reading: Stock price prediction based on financial news using context-specific features. In: 45th Hawaii International Conference on System Science (HICSS), IEEE, pp 1040–1049

  11. Yin J, Tao T, Xu J (2015) A multi-label feature selection algorithm based on multi-objective optimization. In: International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–7

  12. Spolaôr N, Monard MC, Tsoumakas G, Lee HD (2016) A systematic review of multi-label feature selection and a new method based on label construction. Neurocomputing 180:3–15

    Article  Google Scholar 

  13. Xue B, Zhang M, Browne WN, Yao X (2016) A survey on evolutionary computation approaches to feature selection. IEEE Trans Evol Comput 20(4):606–626

    Article  Google Scholar 

  14. Marler RT, Arora JS (2010) The weighted sum method for multi-objective optimization: new insights. Struct Multidiscip Optim 41(6):853–862

    Article  MathSciNet  MATH  Google Scholar 

  15. Zou D, Li S, Kong X, Ouyang H, Li Z (2019) Solving the combined heat and power economic dispatch problems by an improved genetic algorithm and a new constraint handling strategy. Appl Energy 237:646–670

    Article  Google Scholar 

  16. Zou D, Li S, Kong X, Ouyang H, Li Z (2018) Solving the dynamic economic dispatch by a memory-based global differential evolution and a repair technique of constraint handling. Energy 147:59–80

    Article  Google Scholar 

  17. Shen X, Zou D, Duan N, Zhang Q (2019) An efficient fitness-based differential evolution algorithm and a constraint handling technique for dynamic economic emission dispatch. Energy 186:115801

    Article  Google Scholar 

  18. Taradeh M, Mafarja M, Heidari AA, Faris H, Aljarah I, Mirjalili S, Fujita H (2019) An evolutionary gravitational search-based feature selection. Inf Sci 497:219–239

    Article  Google Scholar 

  19. Hichem H, Elkamel M, Rafik M, Mesaaoud MT, Ouahiba C (2019) A new binary grasshopper optimization algorithm for feature selection problem. J King Saud Univ Comput Inf Sci

  20. Storn R, Price K (1997) Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359

    Article  MathSciNet  MATH  Google Scholar 

  21. Rahnamayan S, Wang GG (2008) Solving large scale optimization problems by opposition-based differential evolution (ODE). WSEAS Trans Comput 7(10):1792–1804

    Google Scholar 

  22. Tubishat M, Idris N, Shuib L, Abushariah MA, Mirjalili S (2020) Improved salp swarm algorithm based on opposition based learning and novel local search algorithm for feature selection. Expert Syst Appl 145:113122

    Article  Google Scholar 

  23. Deb K, Jain H (2014) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: Solving problems with box constraints. IEEE Trans Evolut Comput 18(4):577–601

    Article  Google Scholar 

  24. Pereira RB, Plastino A, Zadrozny B, Merschmann LH (2016) Categorizing feature selection methods for multi-label classification. In: Artificial intelligence review, pp 1–22

  25. Dendamrongvit S, Vateekul P, Kubat M (2011) Irrelevant attributes and imbalanced classes in multi-label text-categorization domains. Intell Data Anal 15(6):843–859

    Article  Google Scholar 

  26. Wandekokem E, Varejão F, Rauber T (2010) An overproduce-and-choose strategy to create classifier ensembles with tuned SVM parameters applied to real-world fault diagnosis. Progress Pattern Recognit Image Anal Comput Vis Appl, pp 500–508

  27. Spolaor N, Cherman EA, Monard MC, Lee HD (2013) Relieff for multi-label feature selection. In: Brazilian conference on intelligent systems (BRACIS), IEEE, pp 6–11

  28. Reyes O, Morell C, Ventura S (2015) Scalable extensions of the ReliefF algorithm for weighting and selecting features on the multi-label learning context. Neurocomputing 161:168–182

    Article  Google Scholar 

  29. SpolaoR N, Cherman EA, Monard MC, Lee HD (2013) A comparison of multi-label feature selection methods using the problem transformation approach. Electron Notes Theor Comput Sci 292:135–151

    Article  Google Scholar 

  30. Lin Y, Hu Q, Liu J, Chen J, Duan J (2016) Multi-label feature selection based on neighborhood mutual information. Appl Soft Comput 38:244–256

    Article  Google Scholar 

  31. Safavian SR, Landgrebe D (1991) A survey of decision tree classifier methodology. IEEE Trans Syst Man Cybern 21(3):660–674

    Article  MathSciNet  Google Scholar 

  32. Zhu P, Xu Q, Hu Q, Zhang C, Zhao H (2018) Multi-label feature selection with missing labels. Pattern Recogn 74:488–502

    Article  Google Scholar 

  33. Jiang L, Yu G, Guo M, Wang J (2020) Feature selection with missing labels based on label compression and local feature correlation. Neurocomputing

  34. Cai Z, Zhu W (2018) Multi-label feature selection via feature manifold learning and sparsity regularization. Int J Mach Learn Cybern 9(8):1321–1334

    Article  Google Scholar 

  35. Sun L, Yin T, Ding W, Xu J (2019) Hybrid multilabel feature selection using bpso and neighborhood rough sets for multilabel neighborhood decision systems. IEEE Access 7:175793–175815

    Article  Google Scholar 

  36. Shi M, Tang Y, Zhu X (2019) Mlne: Multi-label network embedding. IEEE Trans Neural Netw Learn Syst

  37. Zhang Y, Dw Gong, Xy Sun, Yn Guo (2017) A PSO-based multi-objective multi-label feature selection method in classification. Sci Rep 7(1):376

    Article  Google Scholar 

  38. Lee J, Seo W (2018) Kim DW (2018) Effective evolutionary multilabel feature selection under a budget constraint. Complexity

  39. Kashef S, Nezamabadi-pour H (2019) A label-specific multi-label feature selection algorithm based on the pareto dominance concept. Pattern Recogn 88:654–667

    Article  Google Scholar 

  40. Bi J, Periaswamy S, Okada K, Kubota T, Fung G, Salganicoff M, Rao RB (2006) Computer aided detection via asymmetric cascade of sparse hyperplane classifiers. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 837–844

  41. Gao D, Vasconcelos N (2005) Integrated learning of saliency, complex features, and object detectors from cluttered scenes. In: IEEE computer society conference on computer vision and pattern recognition, IEEE, vol 2, pp 282–287

  42. Bechikh S, Elarbi M, Said LB (2017) Many-objective optimization using evolutionary algorithms: a survey. In: Recent advances in evolutionary multi-objective optimization, Springer, New York, pp 105–137

  43. Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6(2):182–197

    Article  Google Scholar 

  44. Venkatesh B, Anuradha J (2019) A review of feature selection and its methods. Cybern Inf Technol 19(1):3–26

    MathSciNet  Google Scholar 

  45. Ghavamzadeh M, Lazaric A, Maillard O, Munos R (2010) Lstd with random projections. In: Advances in neural information processing systems, pp 721–729

  46. Sorower MS (2010) A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis, p 18

    Google Scholar 

  47. Sun Z, Zhang J, Dai L, Li C, Zhou C, Xin J, Li S (2019) Mutual information based multi-label feature selection via constrained convex optimization. Neurocomputing 329:447–456

    Article  Google Scholar 

  48. Jungjit S, Michaelis M, Freitas AA, Cinatl J (2013) Two extensions to multi-label correlation-based feature selection: a case study in bioinformatics. In: 2013 IEEE international conference on systems, man, and cybernetics, IEEE, pp 1519–1524

  49. Clare A, King RD (2001) Knowledge discovery in multi-label phenotype data. In: European conference on principles of data mining and knowledge discovery, Springer, pp 42–53

  50. Lindeberg T (2012) Scale invariant feature transform

  51. Vidal-Naquet M, Ullman S (2003) Object recognition with informative features and linear classification. ICCV 3:281–288

    Google Scholar 

  52. Mahdavi S, Rahnamayan S, Deb K (2018) Opposition based learning: a literature review. Swarm Evolut Comput 39:1–23

    Article  Google Scholar 

  53. Rahnamayan S, Tizhoosh HR, Salama MM (2008) Opposition-based differential evolution. IEEE Trans Evol Comput 12(1):64–79

    Article  Google Scholar 

  54. Seif Z, Ahmadi MB (2015) Opposition versus randomness in binary spaces. Appl Soft Comput 27:28–37

    Article  Google Scholar 

  55. Mahdavi S, Rahnamayan S, Mahdavi A (2018) Majority voting for discrete population-based optimization algorithms. Soft Comput, pp 1–18

  56. Velasquez M, Hester PT (2013) An analysis of multi-criteria decision making methods. Int J Oper Res 10(2):56–66

    MathSciNet  Google Scholar 

  57. Bidgoli AA, Rahnamayan S, Mahdavi S, Deb K (2019) A novel pareto-vikor index for ranking scientists’ publication impacts: a case study on evolutionary computation researchers. In: 2019 IEEE congress on evolutionary computation (CEC), IEEE, pp 2458–2465

  58. Zhang ML, Zhou ZH (2007) ML-KNN: A lazy learning approach to multi-label learning. Pattern Recogn 40(7):2038–2048

    Article  MATH  Google Scholar 

  59. Skryjomski P, Krawczyk B, Cano A (2019) Speeding up k-nearest neighbors classifier for large-scale multi-label learning on gpus. Neurocomputing 354:10–19

    Article  Google Scholar 

  60. Tsoumakas G, Katakis I, Vlahavas I (2009) Mining multi-label data. In: Data mining and knowledge discovery handbook, Springer, pp 667–685

  61. Li AD, Xue B, Zhang M (2020) Multi-objective feature selection using hybridization of a genetic algorithm and direct multisearch for key quality characteristic selection. Inf Sci

  62. Bidgoli AA, Ebrahimpour-Komleh H, Rahnamayan S (2020) Reference-point-based multi-objective optimization algorithm with opposition-based voting scheme for multi-label feature selection. Inf Sci 547:1–17

    Article  MathSciNet  Google Scholar 

  63. Zitzler E, Thiele L (1998) Multiobjective optimization using evolutionary algorithms—a comparative case study. In: International conference on parallel problem solving from nature, Springer, pp 292–301

  64. While L, Hingston P, Barone L, Huband S (2006) A faster algorithm for calculating hypervolume. IEEE Trans Evol Comput 10(1):29–38

    Article  Google Scholar 

  65. Wang H, Jin Y, Yao X (2017) Diversity assessment in many-objective optimization. IEEE Trans Cybern 47(6):1510–1522

    Article  Google Scholar 

  66. Woolson R (2007) Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pp 1–3

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Kashan, Kashan, Iran

    Azam Asilian Bidgoli & Hossein Ebrahimpour-komleh

  2. Nature Inspired Computational Intelligence (NICI) Lab, Department of Electrical, Computer, and Software Engineering, Ontario Tech University, Oshawa, Canada

    Shahryar Rahnamayan

Authors
  1. Azam Asilian Bidgoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossein Ebrahimpour-komleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shahryar Rahnamayan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Azam Asilian Bidgoli or Shahryar Rahnamayan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Asilian Bidgoli, A., Ebrahimpour-komleh, H. & Rahnamayan, S. A novel binary many-objective feature selection algorithm for multi-label data classification. Int. J. Mach. Learn. & Cyber. 12, 2041–2057 (2021). https://doi.org/10.1007/s13042-021-01291-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01291-y

Keywords

Search

Navigation