Multi-label learning with label-specific features via weighting and label entropy guided clustering ensemble
Introduction
In traditional supervised learning, each instance has only one associated label. Multi-label learning [1] is different, it deals with the problem where each instance may belong to multiple labels simultaneously. For example, a movie may belong to multiple classes simultaneously, such as action movies and comedies; an image may be annotated with sea and beach; a document may be associated with several topics, including entertainment and sports. It makes multi-label learning much more complicated than single-label learning. Multi-label learning has been involved in many applications, such as text categorization [2], [3], [4], video annotation [5], [6], and music emotion categorization [7], [8]. The task of it is to predict the label set for each test example.
There have been a large number of well-established methods to solve multi-label learning problems. Existing methods can be divided into two categories: problem transformation methods and algorithm adaptation methods. Problem transformation methods transform multi-label learning into other learning scenarios [9], [10]. For example, transform multi-label learning problem into multi-class classification problem or multiple independent binary classification problems. Algorithm adaptation methods modify single-label classification algorithms to deal with multi-label data directly [11], [12], such as KNN and SVM. Due to each instance may be associated with multiple labels simultaneously in multi-label learning, it is important to explore the correlations among labels during the learning process. For example, if the instance contains labels such as “dock”, it is likely to contain the label of “ship”; if the picture contains labels such as “beach”, it is likely to contain the label of “sea”. Many multi-label learning methods have been proposed to exploit the correlations among labels, and they consider label correlations in different ways. For example, some methods consider pairwise relations between labels [13], [14], and some methods consider high-order relations among labels [15], [16]. Besides label correlations, label-specific features also provide a lot of information for multi-label learning. Label-specific features are the most pertinent and discriminative features for each label. For example, texture-based features may be useful in discriminating the label of “desert”; color-based features may be useful in discriminating the label of “sky”. LIFT [17] is the first multi-label learning method based on label-specific features. In LIFT, it performs k-means algorithm [18] on the positive instances and negative instances respectively. The label-specific features are constructed by calculating the distance between the instances and the clustering centers. However, the results of k-means may be unstable, which makes it difficult to explore the underlying properties well. And LIFT ignores the correlations among labels. LIFTACE [19] improves the clustering procedure and considers label correlations. It employs clustering ensemble techniques to make the clustering results more stable and effective. However, it ignores the importance of label vectors and does not explore label correlations when constructing the classification model.
In this paper, a multi-label learning algorithm called LF-LELC is proposed, which considers the importance of label vectors and builds the classification model by considering label correlations. Firstly, LF-LELC employs label entropy to get the information contained in the label vectors [20]. The higher the entropy, the more information the label has. This strategy is flexible since the number of clusters can be adaptively set by label entropy during the clustering process. Secondly, LF-LELC builds the corresponding classification model by exploring label correlations [21]. This strategy can be regarded as an ensemble-style approach, which not only considers label correlations but also improves the performance. Experiments show that our algorithm compares favorably against other comparing algorithms on different types of data sets.
The rest of this paper is organized as follows: Section 2 gives a brief review on the study of multi-label learning. Section 3 describes our proposed algorithm LF-LELC in detail. Section 4 describes our experimental settings, experimental results and discussion. Section 5 concludes this paper.
Section snippets
Related work
Multi-label learning can be divided into two categories: problem transformation methods and algorithm adaptation methods. Problem transformation methods transform multi-label learning into other learning scenarios, e.g., Binary Relevance (BR) [9] and Label Powerset (LP) [10]. Algorithm adaptation methods modify single-label classification algorithms to deal with multi-label data directly, e.g., Multi-Label k-Nearest Neighbor (ML-kNN) [11] and Ranking Support Vector Machine (Rank-SVM) [12].
In
The proposed algorithm
In this paper, we improve LIFTACE proposed by Zhan et al. [19]. Firstly, we employ label entropy to set the number of clusters, which makes the number of clusters more flexible. This strategy is derived from ML-LEC proposed by Zhang et al. [20]. Secondly, we construct the corresponding classification model by exploring label correlations. This strategy is inspired by CTRL proposed by Li et al. [21].
In multi-label learning, let be the instance space with -dimensional and be
Data sets
For each multi-label data set , we use the following notations to denote the properties of it. denotes the number of instances, denotes the number of features, denotes the number of labels, and Domain denotes their application domains. In addition, we list the representations of other multi-label properties as follows:
- •
: Label cardinality measures the average number of associated labels per instance;
- •
: Label density
Conclusion
In this paper, we propose a multi-label learning algorithm named LF-LELC based on label-specific features to exploit label correlations and the information contained in the label vectors. Firstly, it performs clustering analysis on the positive instances and negative instances respectively. The number of clusters is set by the information contained in the label vectors. This strategy considers the importance of label vectors. After that, LF-LELC employs clustering ensemble techniques that
CRediT authorship contribution statement
Chunyu Zhang: Conceptualization, Methodology, Software, Validation, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Zhanshan Li: Validation, Formal analysis, Resources, Writing - review & editing, Supervision, Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to thank Professor Min-Ling Zhang in Southeast University of China for providing the source code of LIFT and other people’s help. Thanks to the reviewers for their valuable comments, so that the paper can be improved. This work is funded by: Natural Science Foundation of Jilin province of China under Grant No. 20180101043JC, Development and Reform Committee Foundation of Jilin province of China under Grant No. 2019C053-9, and Open Research Fund of Key Laboratory of Space
Chunyu Zhang is a master candidate at Jilin University, her interest includes machine learning.
References (33)
- et al.
Learning multi-label scene classification
Pattern Recognition
(2004) - et al.
ML-KNN: A lazy learning approach to multi-label learning
Pattern Recognition
(2007) - et al.
Efficient monte carlo methods for multi-dimensional learning with classifier chains
Pattern Recognition
(2014) - et al.
Multi-label learning with label-specific features by resolving label correlations
Knowledge-Based Systems
(2018) - et al.
Multi-label learning based on label-specific features and local pairwise label correlation
Neurocomputing
(2018) - et al.
Improving multi-label classification with missing labels by learning label-specific features
Information Sciences
(2019) - et al.
A practical approach to feature selection
- et al.
A review on multi-label learning algorithms
IEEE Transactions on Knowledge and Data Engineering
(2014) - et al.
Multilabel neural networks with applications to functional genomics and text categorization
IEEE Transactions on Knowledge and Data Engineering
(2006) - A. McCallum, Multi-label text classification with a mixture model trained by em, in: AAAI workshop on Text Learning,...
Correlative multi-label video annotation
Music emotion recognition by multi-label multi-layer multi-instance multi-view learning
Mining Multi-label Data
Data Mining and Knowledge Discovery Handbook
Cited by (0)
Chunyu Zhang is a master candidate at Jilin University, her interest includes machine learning.
Zhanshan Li is a professor at Jilin University, his interests include constraint optimization and constraint solving, machine learning.