An Ensemble Feature Ranking Algorithm for Clustering Analysis

Abstract

Feature ranking is a widely used feature selection method. It uses importance scores to evaluate features and selects those with high scores. Conventional unsupervised feature ranking methods do not consider the information on cluster structures; therefore, these methods may be unable to select the relevant features for clustering analysis. To address this limitation, we propose a feature ranking algorithm based on silhouette decomposition. The proposed algorithm calculates the ensemble importance scores by decomposing the average silhouette widths of random subspaces. By doing so, the contribution of a feature in generating cluster structures can be represented more clearly. Experiments on different benchmark data sets examined the properties of the proposed algorithm and compared it with the existing ensemble-based feature ranking methods. The experiments demonstrated that the proposed algorithm outperformed its existing counterparts.

Appendices

Appendix 1. Effect of the Random Shuffling Schemes on the Contribution Scores

In this study, we examined the effect of three random shuffling schemes on the computation of the contribution score. The three random shuffling schemes were: (1) random permutation of values within a feature, (2) normal distribution-based random number generation, and (3) uniform distribution-based random number generation. Figure 12 shows the processes used to calculate the contribution scores with different random shuffling schemes that use the same example data as in Fig. 3.

Fig. 12

The processes for contribution score calculation by using such different random shuffling schemes as a random permutation, b normal distribution–based random number generation, and c uniform distribution–based random number generation

In Fig. 12, x₁ is an important feature for clustering analysis, but x₂ is irrelevant. As shown in Fig. 12, both the random permutation and the normal distribution–based random number generation can be used successfully to calculate contribution scores. Conversely, when x₂ is replaced with random numbers generated from the uniform distribution, cluster structures become sparse compared with the original cluster structures (Fig. 12c). Hence, the contribution score of x₂ is much larger than those of other random shuffling schemes. These results demonstrate that the contribution scores obtained by the uniform distribution–based random number generation scheme may lead to inappropriate evaluation of feature importance for clustering analysis.

Appendix 2. Investigation of the k_max Value Determination Methods

Because the ik-means algorithm tends to identify more clusters than are actually present in data sets (Chiang and Mirkin 2010; de Amorim et al. 2016), this algorithm can also be used to determine k_max values for the FRSD algorithm. The ik-means algorithm iteratively identifies a set of anomalous patterns farthest from the center of data sets. If the number of observations in the anomalous pattern exceeds a predefined threshold, the anomalous patterns are considered clusters. Otherwise, these patterns are treated as outliers. (In this study, we set the threshold value as one.) The anomalous patterns are removed from the training data set, and this process is repeated until all observations belong to the cluster or are determined to be outliers. The final number of clusters identified can be used as the k_max values for the FRSD algorithm. Table 6 shows the values of k_max obtained from the ik-means algorithm and Eq.12.

Table 6 k_max values obtained from Eq. 12 ($\min \left(\left\lceil \sqrt{n}\right\rceil, 20\right)$, where n denotes the number of training observations) and the ik-means the algorithm for the FRSD algorithm

Table 6 shows that in all cases the values of k_max obtained from the ik-means algorithm are less than those computed by Eq. 12. Moreover, in Armstrong-2002-v2 and in the brain tumor cases, all observations are considered outliers because this algorithm is vulnerable to high dimensionality (de Amorim and Mirkin 2012). In this study, we executed 20 runs of the FRSD algorithm using each k_max setting method (ik-means and Eq. 12) and reported the mean values and standard deviations of the adjusted Rand indices.

As shown in Fig. 13, in most of the cases, Eq. 12 shows similar or better performance compared with the ik-means algorithm because in high-dimensional data, Eq.12 tends to generate more clusters than the ik-means algorithm. Hence, we suggest using Eq. 12 ($\min \left(\left\lceil \sqrt{n}\right\rceil, 20\right)$) to determine the k_max value for the FRSD algorithm.

Fig. 13

The average classification accuracy of the FRSD algorithm using ik-means and Eq. 12 for a Chen-2002, b Rsinger-2003, c Alizadeh-2000-v3, d lymphoma, and e Garber-2001 cases