Adaptive discriminant analysis for semi-supervised feature selection

Abstract

As semi-supervised feature selection is becoming much more popular among researchers, many related methods have been proposed in recent years. However, many of these methods first compute a similarity matrix prior to feature selection, and the matrix is then fixed during the subsequent feature selection process. Clearly, the similarity matrix generated from the original dataset is susceptible to the noise features. In this paper, we propose a novel adaptive discriminant analysis for semi-supervised feature selection, namely, SADA. Instead of computing a similarity matrix first, SADA simultaneously learns an adaptive similarity matrix S and a projection matrix W with an iterative process. Moreover. we introduce the ${\ell }_{2,p}$ norm to control the sparsity of S by adjusting p. Experimental results show that S will become sparser with the decrease of p. The experimental results for synthetic datasets and nine benchmark datasets demonstrate the superiority of SADA, in comparison with 6 semi-supervised feature selection methods.

Introduction

Feature selection is very important for high-dimensional data analysis because it can remove irrelevant features with slight performance deterioration [1]. With the rapid increase of the data size, obtaining labeled data is often costly [2]. Therefore, to free us from the laborious and tedious data labeling work, only a small set of data samples are expected to be marked with ground truth. At the same time, it is desirable to exploit unlabeled samples during training to ensure the effectiveness of the learned models. The research topics related to this problem such as image annotations and categorizations have become hot spots in many machine learning fields [3], [4]. Thus, it is desirable to develop feature selection methods that can exploit both labeled and unlabeled data. Therefore, the study of “semi-supervised feature selection” has gained increasing attention [5], [6], [7].

Due to the advantages of semi-supervised feature selection, related methods have sprung up in recent years. However, these methods have a shortcoming of measuring features with a ranking criterion without considering the models [8], [9], [10], [11]. Ren et al. proposed a wrapper-type forward semi-supervised feature selection framework [12] that exploits labeled data and unlabeled data for the supervised sequential forward semi-supervised feature selection (SFFS). Xu et al. introduced a discriminative semi-supervised feature selection method based on the idea of manifold regularization, making use of classification margin and geometry of the probability distribution to select the features. However, because of its computational complexity of $O(n^{2.5}/∊)$, where n is the number of objects and $∊$ is a fairly small stopping criterion, their method is time-consuming [13]. To choose the “best so far” feature subset from the streaming features, Wu et al. proposed a novel feature selection method called online streaming feature selection (OSFS) [14]. However, domain knowledge is required in OSFS, and thus, Eskandari et al. chose to use rough sets (RS) to optimize the former model (OS-NRRSAR-SA) [15]. Recently, Zhou et al. further improved the new OS-NRRSAR-SA model with the adapted neighborhood rough set [16]. Chen et al. have also used the rough set to perform feature selection on imbalanced data [17]. Embedded semi-supervised methods are superior to other feature selection methods in many ways because feature selection is set as part of the model training process. Chen proposed a semi-supervised feature selection method RLSR [5] in which a rescaled linear square regression is added to extend the least-squares regression for feature selection. Yuan et al. improved RLSR by introducing an $∊$-dragging technique to enlarge the distances between different classes [18]. In addition to the methods mentioned above, other methods such as semi-supervised feature selection via spline regression [19], ensemble feature selection [20], [21], [22], and parallel feature selection [23], [24] have also been developed. For most feature selection methods, a pair-wise similarity matrix is constructed from the original data and then set as fixed during the subsequent feature selection process. Researchers have designed several metrics to quantify the similarity between features based on some powerful tools like mutual information and graphs [25], [26]. However, as pointed in [27], such a similarity matrix may lose the inner class structure on the multimodal data in which samples in some classes form several separate clusters, and often mislead the feature selection methods into recovering the wrong local structure since it is easily affected by noise features.

Sparsity commonly exists in real-world data, thus, sparse learning has become a key component in feature selection. Shi et al. consider the superiority of $l_{2,p}$-norm as well as its non-Lipschitz continuity and claim the effectiveness of $l_{2,1-2}$-norm. Moreover, they apply CCCP and ADMM to solve the non-convex problem [28]. Zhang et al. draw the conclusion that for $l_{2,p}$-norm, a smaller p leads to higher performance. They also discuss the situation where $p\to 0$ and proposed two algorithms to optimize the discrete feature selection problem [29].

Recently, Chen et al. have proposed a LAP framework for both labeled and unlabeled data [30]. Instead of computing a fixed similarity matrix prior to performing feature selection, LAP learns an adaptive similarity matrix $\mathbf{S}$ and a projection matrix $\mathbf{W}$ simultaneously with an iterative method. Based on this method, we extend it to semi-supervised feature selection tasks by proposing a semi-supervised adaptive discriminant analysis (SADA). As an extension to LAP, SADA can learn better a similarity matrix by weakening the effect of noisy features on the similarity computing and deal with multimodal data [27] by investigating local structure in the data. The main contributions of our work include:

• We rewrite $\left|\right|W^T(x_i-x_j)|{|}_2$ as $\left|\right|W^T(x_i-x_j)|{|}_{2,p}^p$ by introducing the ${\ell }_{2,p}$ norm to control the sparsity of S by adjusting p, which can be used to adaptively preserve the locality. Experimental results show that S will become sparser with the decrease of p.

• We take both labeled and unlabeled data into account to better preserve the locality in the semi-supervised scenario.

• Comprehensive experiments on 9 benchmark datasets show the superior performance of the proposed approach in comparison with 6 semi-supervised feature selection methods.

The rest of this paper is organized as follows. Section 2 presents the notations, and Section 3 surveys the existing semi-supervised feature selection methods. The semi-supervised feature selection method, SADA, is proposed in Section 4. We present the experimental results and analysis in Section 5. The conclusions and directions for future work are provided in Section 6.