The k-means Algorithm: A Comprehensive Survey and Performance Evaluation

Abstract

The k-means clustering algorithm is considered one of the most powerful and popular data mining algorithms in the research community. However, despite its popularity, the algorithm has certain limitations, including problems associated with random initialization of the centroids which leads to unexpected convergence. Additionally, such a clustering algorithm requires the number of clusters to be defined beforehand, which is responsible for different cluster shapes and outlier effects. A fundamental problem of the k-means algorithm is its inability to handle various data types. This paper provides a structured and synoptic overview of research conducted on the k-means algorithm to overcome such shortcomings. Variants of the k-means algorithms including their recent developments are discussed, where their effectiveness is investigated based on the experimental analysis of a variety of datasets. The detailed experimental analysis along with a thorough comparison among different k-means clustering algorithms differentiates our work compared to other existing survey papers. Furthermore, it outlines a clear and thorough understanding of the k-means algorithm along with its different research directions.

1. Introduction

The advancements in computing, along with the rapid growth and availability of data repositories, have often emphasized the task of gaining meaningful insights from the data. This encourages taking appropriate measures based on knowledge discovery approaches. Machine Learning (ML) can be broadly classified into two: supervised and unsupervised learning. In the case of supervised learning, a function is learned that maps a given input to an output based on the available input–output pairs. These learning algorithms thus require the availability of labeled data with the desired output value [1]. The availability of labeled data represents an ideal scenario; however, such datasets are often expensive and challenging to obtain. For instance, in the intrusion detection domain, zero-day attacks are rare instances and obtaining labels for them is expensive. Hence, when the labels of the datasets are unavailable, unsupervised learning approaches are typically used [2]. Under such a learning framework, the algorithm has no prior knowledge of the true labels of the dataset and tries to draw inferences from the dataset itself. A more recent and popular set of algorithms referred to as semi-supervised learning consists of algorithms that lie in between supervised and unsupervised learning. Such a learning framework makes use of labeled and unlabeled data for better inference. Existing research on semi-supervised learning corroborates that adding a skimpy amount of labeled data together with a large amount of unlabeled data produces considerable improvements in the accuracy of the predictive model. This manuscript primarily deals with different variations of the k-means algorithm, which falls under the family of unsupervised learning. Therefore, this paper will focus only on unsupervised learning algorithms.

Clustering algorithms exploit the underlying structure of the data distribution and define rules for grouping the data with similar characteristics [3]. This process results in the partition of a given dataset according to the clustering criteria without any prior knowledge about the dataset. In an ideal clustering scenario, each cluster consists of similar data instances that are quite dissimilar from the instances in other clusters. Such a dissimilarity measure relies on the underlying data and objective of the algorithm. Clustering is central to many data-driven applications and is considered an interesting and important task in machine learning. It is also studied in statistics, pattern recognition, computational geometry, bioinformatics, optimization, image processing, and in a variety of other fields [4,5,6,7]. A plethora of clustering techniques have been invented in the last decade, which are being applied in a wide range of application domains [8].

Table 1. Applications of variants of the k-means algorithm in different application domains.

Reference	Application	Algorithm
[10]	Face detection	Symmetry-based version of k-means (SBKM).
[11]	Mobile storage positioning	Potential k-means.
[12]	Load pattern	Hierarchical k-means (H-Kmeans).
[13]	Wireless sensor networks	Distributed k-means and fuzzy c-means.
[14]	Partial multiview data	Weighted k-means.
[15]	Mobile health	k-means implemented with CORDIC.
[16]	Endpoint detection	k-means for realtime detection.
[17]	Big data	Privacy preserving k-means.
[18]	Multiview data	k-means.
[19]	Wind power forecasting	k-means with bagging neural network.
[20]	Social tags	k-means based on latent semantic analysis.
[21]	Sensing for IGBT current	k-means with neural network.
[22]	Image segmentation.	Kernel k-means Nystrom approximation.
[23]	Image compression	k-means cuckoo optimization.
[24]	Sound source angle estimation.	Neural network based on global k-means.
[25]	Shape recognition	Fuzzy k-means clustering ensemble (FKMCE).
[26]	Signal processing	Compressive k-means clustering (CKM).
[27]	Text processing	Vanilla k-means.
[28]	High dimensional data processing	Fast adaptive k-means (FAKM).
[29]	Computational complexity	Multiple kernel k-means with late fusion.
[30]	Image processing	A hybrid parallelization of k-means algorithm.
[31]	Adaptive clustering	Fuzzy k-means with S-distance.
[32]	DDoS detection	Semi-supervised k-means algorithm with hybrid feature.
[33]	Optimization	Non alternating stochastic k-means.
[34]	Data Summarization	Modified x-means.

shows the recent applications on k-means clustering [9] in different application domains.

This survey studies the problems of and solutions to partition-based clustering, and more specifically the widely used k-means algorithm [9], which has been listed among the top 10 clustering algorithms for data analysis [35]. Due to their popularity and ease of use, k-means clustering methods are being used in conjunction with deep learning for tasks such as image segmentation and handwriting recognition [36]. A more recent work in [37] used a fully connected deep convolution neural network along with k-means and performed pixel matching between a segmented image and a convoluted image. This overcomes the problem that exists when useful information from images is lost by repeated convolution of the images [36]. Although the k-means clustering algorithm itself performs well with compact and hyper-spherical clusters, we are interested in highlighting its limitations and suggesting solutions. The primary focus is given to two unavoidable problems of the k-means algorithm: (i) assignment of centroids and number of clusters and (ii) ability to handle different types of data.

Although researchers have proposed variants (Figure 1) of k-means algorithms to overcome these impediments, they are however domain specific and do not generalize well. For example, a k-means algorithm which can handle categorical data might perform poorly because of the initialization process used.

Table 2. Comparison with existing surveys.

Survey	Initialization	Data Types	Applications	Experiments
Yang [38]	✓	×	×	×
Filippone [39]	✓	×	×	×
Rai [40]	✓	×	×	×
This paper	✓	✓	✓	✓

outlines a comparative analysis of existing surveys on clustering and highlights the contributions of this paper. The key contributions of this paper are listed below.

• Existing solutions of the k-means algorithm along with a taxonomy are outlined and discussed in order to augment the understanding of these variants and their relationships.
• This research frames the problems, analyses their solutions and presents a concrete study on advances in the development of the k-means algorithm.
• Experiments are performed using the improved k-means algorithms to find out their effectiveness using six benchmark datasets.

Paper Roadmap

Our paper is structured as follows. Section 2 and Section 3 address the problems of the k-means algorithm and review the existing solutions for them. Section 4 showcases an experimental analysis of the representative algorithms (from Section 2 and Section 3) on six benchmark datasets widely used by the machine learning and data mining research community. Section 5 concludes the paper, summarizing the key insights regarding the k-means algorithm.

2. k-means Variants for Solving the Problem of Initialization

The k-means algorithm depends on the value of k; which always needs to be specified in order to perform any clustering analysis. Clustering with different k values will eventually produce different results. Different initialization problems that were analyzed in recent studies did not consider the problem where the algorithm only converges to a poor local minima. In [41], an alternative approach was adopted to prevent the k-means algorithm from being easily affected by noise and outlier values. The authors presented a modified k-means algorithm based on self-paced learning theory. Self-paced learning theory is used in order to select competing training subset that helps the k-means algorithm to build an initial cluster model. The generalization ability of the algorithm is then improved by subsequently adding training subsets found by self-paced learning until the model reaches an optimal performance or all the training samples have been used. The authors proposed this algorithm and demonstrated its performances on several real datasets.

The authors in [42] proposed an improvement to the vanilla k-means algorithm that prevents it from getting stuck to a local minima. The improved algorithm incorporates cuckoo search along with the k-means algorithm. The method incorporates a modified cuckoo search algorithm, which helps to reach a better solution since the search step size factor is being changed. Overall, the proposed algorithm is robust as well as fast in reaching a near-optimal solution.

In [43], the authors discussed criteria concerning the stability and the performance of the k-means. Theoretical results of algorithm stability were proven as the number of instances approaches infinity. The authors in [44] analyzed the stability of the k-means clustering algorithm in a more practical scenario, the parameter (cluster number) is chosen by stability-based model selection. The primary focus was towards drawing random sample points to explain the effect on the stability of the algorithm.

An initial estimation method for computing the covariance matrix for the k-means algorithm using Mahalanobis distance was proposed in [45]. It involves finding a group of points comprised of neighbors with a high density that represent the centroids of the selected clusters. These provide an approximate estimate of the covariance matrix, which is updated successively using the proposed algorithm.

Ball et al. [46] proposed the ISODATA algorithm to estimate k by parameter tweaking. A similar strategy is also adopted in adaptive resonance theory [47], which generates new clusters [48]. Since this approach requires a predefined threshold, it is generally avoided.

In x-means clustering [49], the BIC (Bayesian information criterion) or “Schwarz criterion” [50,51] is utilised to find the number of k clusters. For a given set of data, a family of different models initialized with different k values is used to compute the posterior probability distribution. These distributions are then used to find the score of the different models. This algorithm was utilized in identifying anomalous patterns in [52,53].

Bradley et al. [54] outlined methods for identifying a more fine grained starting condition that relies on the estimation of the distribution modes. The algorithm initially chooses small samples from the dataset and applies k-means clustering. These temporary sets are then further clustered using the k-means to find better initialization. This algorithm’s computational complexity is significantly lower compared to vanilla k-means and, more importantly, is scalable to large datasets. Figure 2 displays the methodology for obtaining refined initial points.

In [55], the author empirically compared four initialization methods. These are as follows: (i) RANDOM, (ii) FA [56], (iii) MA [9], and (iv) KA [57]. The RANDOM method divides the dataset randomly into k number of clusters and is one of the most popular initialization methods. The FA approach [56] randomly chooses k instances from the given dataset and assigns the remaining instances to the nearest cluster centers. The MA approach [9] randomly selects k instances of the dataset, similar to basic k-means. The KA approach [57] provides a method where the initial clustering is done by selecting representative instances successively until k instances are found. The first representative instance is the most centrally located in the dataset. The remaining representative instances are chosen based on the heuristic that presents rules of choosing instances with higher numbers compared to others. Based on their experimental analysis, the RANDOM and KA resulted in superior performance compared to the other two. Recently, in [58], a co-clustering algorithm is proposed for dealing with sparse and high dimensional data that outperforms the basic k-means.

These solutions can only handle numerical data. Hence such methods would not be suitable for datasets consisting of mixed attributes, for instance, categorical and binary. In the next section, we discuss the research progress dealing with mixed types of data for clustering.

3. k-means Variants for Solving the Problem of Data Issue

In many application domains, datasets include attributes that are of mixed data types [59]. Therefore, a distance calculation scheme is necessary that is able to compute distances between instances having mixed types of data. This section discusses existing research on k-means to find similarity or distance metrics between categorical attributes. Although the algorithms discussed in the previous section solve the initialization problem, they still rely on numerical data for distance calculation. Therefore, it is necessary to explore the distance calculation mechanism for the k-means clustering algorithm and our investigation as follows.

Wang et al. [60] introduced an extended form of the fuzzy k-means (xFKM) algorithm for datasets with non-numerical attributes, such as categorical data. The centroids constitute an extended form to retain as much clustering information as possible. A computational cost comparison was made among the k-means, fuzzy k-means and xFKM. They showed that the xFKM is most suitable for datasets with categorical attributes, where the attributes vary in a medium and acceptable range of diverse values.

Amir et al. [61] offered a cost function and distance measure for clustering datasets with mixed data (datasets with numerical and categorical data) based on co-occurrences of values. In [62], a kernel function based on “hamming distance” [62] was proposed for embedding categorical data. The kernel-k-means provides an add-on to the k-means clustering that is designed to find clusters in a feature space where distances are calculated via kernel functions.

Huang et al. in [63] provided a variant of the k-means algorithm, where a dissimilarity measure is used as a metric. The algorithm combines the k-means and k-modes clustering for data having mixed attributes. In [64], the convergence of different versions of the modified k-modes algorithms was analyzed. The authors proved that the modified algorithms fail to converge to a local minima without degrading the original k-mode algorithm. To handle this issue, the authors presented two algorithms, MKM-NOF and MKM-NDM, that apply different methods for representing clusters by weighted cluster prototypes.

In [65], an “ellipsoidal k-means” algorithm was proposed that extends the “spherical k-means” algorithm for selecting features in a sparse and high-dimensional dataset. In [66], an entropy weighting k-means (EWKM) clustering algorithm was presented for clustering sparse datasets with high dimensions. The algorithm reduces the dispersion within the cluster and increases the negative weight entropy.

Existing solutions for handling mixed data certainly enhanced the capability of the k-means; however, as the solutions do not address the initialization problem, several issues with k-means still remain. Among many proposed measures [67,68,69], the overlap measure is considered to be the easiest and most computationally efficient method to find similarity between two categorical attributes [70]. One of the shortcomings of this method is inability to differentiate between the distinct attribute values, however, focuses on whether two categorical attributes attain equal values or not. However, after a thorough experimental analysis in [71] on a number of datasets, the authors came to the conclusion that “No single measure is always superior or inferior. This is to be expected since each dataset has different characteristics”. Hence, the overlap measure (for categorical attributes) and Euclidean distance (for numerical attributes) can be combined into a mixed distance measure, as also used in [6,72].

4. Performance Evaluation of k-means Representative Variants

We performed an experimental analysis to investigate different versions of the k-means algorithms for different datasets, especially for the initialization and mixed-data-type problems (GitHub Link: shorturl.at/CR245). All datasets are available in te UCI machine learning repository [73].

Table 3. Summary of datasets.

Dataset	Summary
Cleveland Heart Disease	Widely used by machine learning researchers. The goal is to detect the presence of heart disease in a patient.
KDD-Cup 1999(10%)	Contains standard network traffic that contains different types of cyber attacks simulated in a military network.
Wisconsin Diagnostic Breast Cancer	Includes features calculated from the images of fine needle aspirate of breast mass.
Epileptic Seizure Recognition	Commonly used for feature epileptic seizure prediction.
Credit Approval	Contains a mix of attributes, which makes it interesting to be used with k-means for mixed attributes.
Postoperative	Contains both categorical and integer values. The missing values are replaced with an average.

briefly summarizes the datasets. The next subsections discuss the evaluation metrics used, results analysis and, last but not least, the analysis of computational complexity.

4.1. Metrics Used for Experimental Analysis

To evaluate the performance of different algorithms, the following metrics [1,2,3] were chosen:

• Accuracy: This measure outlines the extent to which the predicted labels are in agreement with the true labels. The predicted labels correspond to the class labels where new instances are clustered. The accuracy is calculated by Equation (1).

  $$Accuracy={\displaystyle \frac{\mathrm{Correctly}\mathrm{identified}\mathrm{class}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{class}}}\times 100$$

  (1)
• Adjusted rand index (ARI): Provides a score of similarity between two different clustering results of the same dataset. For a given set S consisting of $\alpha$ elements and r subsets and two partitions $Y=\{Y_1,Y_2,\cdots ,Y_b\}$ and $X=\{X_1,X_2,\cdots ,X_c\}$, the overlap between the two partitions can be summarized as follows:

  $$\begin{array}{cccccc}& Y_1\hfill & Y_2\hfill & \cdots \hfill & Y_b\hfill & \mathrm{Sums}\hfill \\ X_1\hfill & {\alpha }_{11}\hfill & {\alpha }_{12}\hfill & \hfill & {\alpha }_{1b}\hfill & r_1\hfill \\ X_2\hfill & {\alpha }_{21}\hfill & {\alpha }_{22}\hfill & \hfill & {\alpha }_{2b}\hfill & r_2\hfill \\ ⋮\hfill & ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill & ⋮\hfill \\ X_c\hfill & {\alpha }_{c1}\hfill & {\alpha }_{c2}\hfill & \cdots \hfill & {\alpha }_{cb}\hfill & r_c\hfill \\ \mathrm{Sums}\hfill & s_1\hfill & s_2\hfill & \cdots \hfill & s_b\hfill & \hfill \end{array}$$

  The adjusted rand index (ARI) is then calculated by Equation (2):

  $$ARI={\displaystyle \frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{{\alpha }_{ij}}{2}\right)-\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{r_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{s_j}{2}\right)\right]/\left(\genfrac{}{}{0pt}{}{\alpha }{2}\right)}{\frac{1}{2}[{\sum }_i\left(\genfrac{}{}{0pt}{}{r_i}{2}\right)+{\sum }_j\left(\genfrac{}{}{0pt}{}{s_j}{2}\right)]-\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{r_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{s_j}{2}\right)\right]/\left(\genfrac{}{}{0pt}{}{\alpha }{2}\right)}}$$

  (2)

  The ARI score is adjusted to have values between 0 and 1 to represent scores for random and perfect clustering, respectively.

4.2. Results

Experimental analysis with k-means [9], x-means [49], constrained k-means [54], k-prototype [63], and kernel k-means [74] was performed, and a performance comparison in terms of the metrics described above was made. For each of the experiments, a 10 fold cross validation scheme was used to split the dataset, and the reported results consist of average and standard deviation of the scores of these metrics across 10 validation folds.

Table 4. Comparison of different algorithms addressing initialization issue.

Metric	k-means	Constrained k-means	x-means
Wisconsin Diagnostic Breast Cancer
Accuracy	$0.223\pm 0.310$	$0.596\pm 0.406$	$0.086\pm 0.042$
ARI	$0.690\pm 0.134$	$0.682\pm 0.13$	$0.683\pm 0.128$
KDD Cup 1999
Accuracy	$0.195\pm 0.077$	$0.118\pm 0.087$	$0.045\pm 0.034$
ARI	$0.004\pm 0.007$	$0.107\pm 0.059$	$0.085\pm 0.169$
Epileptic Seizure
Accuracy	$0.101\pm 0.060$	$0.099\pm 0.012$	$0.102\pm 0.053$
ARI	$0.005\pm 0.001$	$0.002\pm 0.001$	$0.002\pm 0.002$

addresses the algorithms that deal with initialization issues, while

Table 5. Comparison of k-prototype and Kernel-k-means algorithm datasets with mixed data types.

Metric	k-prototype	Kernel-k-means
Credit Approval
Accuracy	$0.456\pm 0.061$	$0.437\pm 0.283$
ARI	$0.004\pm 0.005$	$0.044\pm 0.092$
Cleveland Heart Disease
Accuracy	$0.462\pm 0.043$	$0.590\pm 0.080$
ARI	$0.003\pm 0.001$	$0.017\pm 0.041$
Post Operative
Accuracy	$0.462\pm 0.043$	$0.590\pm 0.080$
ARI	$0.003\pm 0.001$	$0.017\pm 0.041$

encompasses mixed data type oriented methods.

Table 4 summarizes the mean and the standard deviation across five validation folds and compares the performance between regular k-means, x-means, and constrained k-means algorithms in terms of different metrics for the Wisconsin Diagnostic Breast cancer, KDD Cup 1999 (10%) and Epileptic Seizure datasets. In Figure 3, it is shown that the algorithm performance varied with different datasets. For example, the k-means performed best with the KDD Cup datasets and x-means had the worst accuracy. However, the x-means performed better than the other two in the Epileptic dataset. The constrained-k-means performed best in the Wisconsin dataset. In terms of ARI score, the constrained-k-means seemed to perform in a consistent manner. The key takeaway from these results is that there is no algorithm that will provide a consistent solution regardless of the dataset.

Table 5 compares the performance between the k-prototype and kernel-k-means algorithms. These algorithms are suitable for datasets that consist of mixed attributes. In Figure 4, it is reflected that the k-prototype performed better using the Credit Approval and Post Operative datasets when considering the accuracy of clustering. However, kernel-k-means performed best using the Cleveland Heart Disease dataset. In terms of ARI score comparison, the kernel-k-means algorithm consistently performed better than k-prototype using all three datasets containing mixed data.

4.3. Computational Complexity Analysis

In addition to the comparison of these variants of k-means algorithms in terms of different metrics on different datasets, a time and space complexity comparison of these algorithms was also carried out. Both the time and space complexities of these algorithms depend on the size n of the datasets. Finding an optimal solution for the k-means algorithm is hard in the Euclidean space for both the binary and multi-class clustering problems. The regular k-means algorithm have a time complexity of $\mathcal{O}\left(n^2\right)$, where n is the size of the input data. However, it can be optimized to be linear and be in the order of $\mathcal{O}\left(n\right)$ using certain heuristics mentioned in [75,76]. By contrast, the time complexity for constrained-k-means is of the order $\mathcal{O}\left(kn\right)$, where k represents the amount of clusters. It reflects that the constrained-k-means has a lower time complexity than the regular k-means when the dataset size grows. The x-means algorithm, on the other hand, was mainly proposed in order to address the scalability issue of the regular k-means algorithm for large datasets, since x-means involves a progressive increase in the number of clusters within a user-supplied range ($k_{min},k_{max})$. The time complexity of the x-means algorithm is therefore in the order of $\mathcal{O}(nlog{k}_{max})$. $k_{max}$ is the maximum number clusters possible in a dataset. The space complexity for all these k-means variants is $\mathcal{O}\left(\right(n+k\left)d\right)$, where d is the number of features in a dataset.

Table 6. Complexities of different variants of k-means.

Complexity	k-means	Constrained-k-means	x-means
Time	$\mathcal{O}\left(n^2\right)$	$\mathcal{O}\left(kn\right)$	$\mathcal{O}(nlog{k}_{max})$
Space	$\mathcal{O}\left(\right(n+k\left)d\right)$	$\mathcal{O}\left(\right(n+k\left)d\right)$	$\mathcal{O}\left(\right(n+k\left)d\right)$

shows the complexities of the different clustering algorithms.

5. Conclusions

In a wide range of application domains, data analysis tasks heavily rely on clustering. This paper focused on the popular k-means algorithm and the issues of initialization and inability to handle data with mixed types of features. Unlike other review or survey papers, this paper contains both a critical analysis of the existing literature and an experimental analysis on half a dozen benchmark datasets to demonstrate the performances of different variants of k-means. The experimental analysis divulged that there is no universal solution for the problems of the k-means algorithm; rather each of the existing variants of the algorithm is either application-specific or data-specific. Our future research will focus on developing a robust k-means algorithm that can address both problems simultaneously. This paper will also help the data mining research community to design and develop newer types of clustering algorithms that can address the research issues around Big Data [34,72].