Reliability-based fuzzy clustering ensemble

Abstract

In the clustering ensemble the quality of base-clusterings influences the consensus clustering. Although some researches have been devoted to weighting the base-clustering, fuzzy cluster level weighting has been ignored, more specifically, they did not pay attention to the role of cluster reliability in the fuzzy clustering ensemble. In this paper, we propose a new fuzzy clustering ensemble framework without access to the features of data-objects based on fuzzy cluster-level weighting. The reliability of each fuzzy cluster is computed based on estimation of its unreliability, and is considered as its weight in the ensemble. The unreliability of fuzzy clusters is estimated by applying the similarity between fuzzy clusters in the ensemble based on an entropic criterion. In our framework, the final clustering is produced by two types of consensus functions: (1) a reliability-based weighted fuzzy co-association matrix is constructed from the base-clusterings and then, a single traditional clustering such as hierarchical agglomerative clustering or K-means is applied over the matrix to produce the final clustering. (2) a new graph based fuzzy consensuses function. The graph based consensus function has linear time complexity in the number of data-objects. Experimental results on various standard datasets demonstrated the effectiveness of the proposed approach compared to the state-of-the-art methods in terms of evaluation criteria and clustering robustness.

Introduction

Clustering is the process of partitioning a set of data-objects (samples) into some (K) subsets of data-objects based on a similarity (distance) measure, where the data-objects in each subset are more similar to one another, and more separate than other subsets of data-objects. Each subset in the mentioned definition is usually referred to as a cluster. All clusters together are named a clustering. Based on the relationship of each data-object to the clusters, the clustering algorithms can also be divided into crisp and fuzzy clustering algorithms. In crisp clustering a data object definitely belongs to one cluster. In fuzzy clustering, data-objects are assigned to every cluster with a membership degree. Crisp clustering is a special case of fuzzy clustering, in which the membership degree of a data-object belonging to a cluster equals to one and its membership degree belonging to the other clusters is zero.

In general, in data clustering context, various clustering algorithms have emerged, each uses a different similarity criterion. Therefore, they have different objective functions. All these methods are heavily dependent on dataset; in other words, there is no clustering algorithm that can learn every dataset [1]. Hence, data clustering with the help of an ensemble of clusters has been proposed as a technique for resolving the aforementioned problems in recent years by researchers [2], [3], [4], [5]. This technique is named clustering ensemble. The main aim of clustering ensemble is to search for a likely better and more stable result with the aggregation of the information extracted from multiple clusterings (also called base-clusterings or members) [6], [7]. The better and more robust result that is extracted from base-clusterings is named consensus clustering (which in this research is also referred to as the final clustering) [6], [8].

In summary, as observed in Fig. 1, a clustering ensemble consists of the following two phases [6]: (1) Base-clustering generation phase: Produce base-clusterings through single clustering algorithms (in this study single clustering is used versus ensemble clustering). (2) Base-clustering consolidation: In this phase the base-clusterings generated in phase 1 must be combined in order to generate the final clustering, which is the objective of this phase. This consolidation is done through a consensus function. It is worth mentioning that this paper focuses on this phase, and proposes a co-association based fuzzy clustering ensemble method and a graph based fuzzy clustering ensemble method.

Despite the greater generality of fuzzy clustering compared to crisp clustering, researches in fuzzy cluster ensemble are still in the initial stages and there exist relatively few approaches for this field. Some of the existing fuzzy cluster ensemble methods convert fuzzy clusters into hard clusters at first, and then compute the final clustering through the hard consensus functions, which causes the loss of uncertainty information. Therefore, proposing an efficient fuzzy consensus clustering from multiple fuzzy base-clusterings remains a challenging issue.

In the ensemble of voters (learners) assigning weight to each learner based on its quality can be an effective mechanism to improve the result of the ensemble. The process of weighting to learners in an ensemble of learners can be optimally set if the accuracy of each learner is known [4]. But if the learners are of type clustering algorithms, the accuracy for learner is meaningless [9]. So, the quality of clustering that is obtained by clustering algorithm can be used as an approximation for its (clustering algorithm) accuracy. In summary, the quality of the base-clusterings highly affects the consensus clustering (final clustering) obtained by the ensemble. In other words, low-quality base-clusterings may have a negative influence on the consensus results. Some researchers investigated the quality-evaluation and weighting of the base-clusterings to improve the consensus clustering quality [10], [11], [12]. For a method that used weighting mechanism in fuzzy clustering ensemble, we can refer to a paper by Berikov [13]. However, this approach assumed that all of the clusters in the same base-clustering have the same reliability; They typically treat each base clustering as an individual and assign a weight to them regardless of the diversity of the clusters inside [10], [11], [12], [13]. In this research, reliability is defined as the quantity of certain knowledge of the ensemble about the cluster and is computed by the accretion amount of that cluster by the ensemble. Briefly, in the aforementioned papers weighting is considered at the clustering level not in the cluster level. But due to the inherent complexity of real-world datasets, the different clusters in the same clustering may have different reliability. Hence, it is necessary to consider the local diversity of ensembles (quality of the clusters in the ensemble) and deal with the different reliability of clusters. Although Zhong et al. investigated the reliability of crisp clusters by considering the Euclidean distances between data objects in clusters [14], this method is not operational for fuzzy clustering, in addition, it requires access to the original data features, and its efficacy relies heavily on the data distribution of the dataset, while in the general formulation of cluster ensemble, there exists no access to the original data features. Therefore, without the need to access the data features or rely on specific assumptions made on data distribution, the key question here is how to measure the reliability of fuzzy clusters and weight them accordingly to enhance the accuracy and robustness of the consensus clustering. In other words, the problem that must be solved here is how to compute the reliability of each fuzzy cluster as a fuzzy cluster quality measure and incorporate it into a weighting structure for boosting the consensus clustering.

In light of this, we propose a new fuzzy clustering ensemble framework based on ensemble-driven cluster reliability and local weighting strategy framework; we assign a weight to each cluster based on its reliability value. The contributions of this article are as follows:

• A method is proposed to estimate the unreliability of fuzzy clusters in relation to a clustering by considering the membership degree of all data-objects to the clusters by applying an entropic criterion, which requires no access to the original data features.

• A reliability-driven cluster indicator is proposed to measure the reliability of the fuzzy clusters in the ensemble and consider it as the weight of the fuzzy clusters in the ensemble.

• A method is proposed to compute the reliability-based fuzzy co-association matrix in the fuzzy clustering ensemble.

• Applying three single clustering algorithms on the obtained co-association matrix and showing their effects on the quality of the proposed approach on a variety of datasets.

• A reliability-based graph consensus function is proposed whose time complexity is linear in the number of data-objects.

• Extensive experiments performed on a variety of datasets indicate that this proposed fuzzy clustering ensemble approach performs better than the state-of-the-art approaches in terms of clustering quality.

The rest of this work is structured as follows: Sec. 2 presents a review of related work. The formal background knowledge about ensemble clustering is presented in Sec. 3. The proposed fuzzy clustering ensemble approach is explained in Sec. 4. We show the experimental results in Sec. 5 and the conclusion and future work are presented in Sec. 6.