A simulated annealing algorithm with a dual perturbation method for clustering

Highlights

• Existing partitional clustering algorithms still settle upon local optima.

• We propose a new simulated annealing algorithm with two perturbation methods.

• We compare our algorithm with existing simulated annealing clustering algorithms.

• We show our new algorithm produces clusters of higher quality more consistently.

Abstract

Clustering is a powerful tool in exploratory data analysis that partitions a set of objects into clusters with the goal of maximizing the similarity of objects within each cluster. Due to the tendency of clustering algorithms to find suboptimal partitions of data, the approximation method Simulated Annealing (SA) has been used to search for near-optimal partitions. However, existing SA-based partitional clustering algorithms still settle to local optima. We propose a new SA-based clustering algorithm, the Simulated Annealing with Gaussian Mutation and Distortion Equalization algorithm (SAGMDE), which uses two perturbation methods to allow for both large and small perturbations in solutions. Our experiments on a diverse collection of data sets show that SAGMDE performs more consistently and yields better results than existing SA clustering algorithms in terms of cluster quality while maintaining a reasonable runtime. Finally, we use generative art as a visualization tool to compare various partitional clustering algorithms.

Introduction

The ever-growing abundance of data prompts further research into data mining. An important and well-studied technique within the field of data mining is clustering, a type of unsupervised machine learning that involves grouping data based on predefined attributes. Clustering is applicable to a vast array of fields ranging from statistics to engineering to psychology [1]. For example, this tool has applications in medical fields through image segmentation [2] as well as business through market analysis [3].

Clustering is an NP-hard problem [4] that is generally solved by assigning $N$data points in $D$dimensions to $K$clusters with the goal of optimizing these clusters based on a given criterion. There are two main categories of clustering: hierarchical and partitional [5]. In hierarchical clustering, objects in the data set are grouped into a nested hierarchy of clusters. Partitional clustering involves dividing the objects into groups simultaneously without creating a nested structure.

Researchers have also investigated a third category of clustering, density-based clustering, which builds clusters based on the density of data points in a given region of the $D$-dimensional space. This class of algorithms has the benefit of reduced sensitivity to noise and cluster shape, but this strategy generally does not perform well for high-dimensional data [6]. Examples of algorithms that fully or partially employ a density-based strategy include a critical distance-based algorithm [7] and a gravity-center algorithm [8].

This paper will focus on partitional clustering algorithms. The most popular approach to partitional clustering is center-based clustering. Center-based, iterative clustering algorithms follow these two general steps:

• Centers are initialized

• Until convergence, every point’s membership to each center and every point’s weight are recomputed and used to recalculate the centers

The K-means algorithm (KM) is the best-known center-based clustering algorithm. It finds $K$centers in $D$dimensions with the goal of maximizing the similarity of the elements within each cluster based upon the provided attributes of each object. To accomplish this goal, the algorithm attempts to minimize the sum of the distances between each point and its closest center, which is the Sum of Squares Error (SSE). Given that $X=\left\{x_i\right|i=1,\dots ,N\}$represents the $N$data points and $M=\left\{m_l\right|l=1,\dots ,K\}$represents the current $K$centers, the performance function for K-means is shown below:

$$Per{f}_{KM}(X,M)=\sum _{i=1}^{N}MIN\{\parallel x_i-m_l{\parallel }^2|l=1,\dots ,K\}$$

The distance-based approach of KM requires quantitative data, but researchers have produced variants of KM to perform clustering on mixed numerical and categorical data [9]. KM has a hard membership function, meaning that each point belongs to only one cluster (in this case its nearest cluster) as opposed to soft membership, where a proportion of each point belongs to multiple centers [10]. Additionally, every point in KM has the same weight. In KM, only the points close to a given center are considered in the recalculation of that center’s position; as a result, KM is sensitive to initialization and converges upon a local minimum.

There are also studies aimed at finding the optimal value of $K$for center-based clustering algorithms such as Pelleg and Moore’s study [11]. Guan et al. [12] proposed the K-means+ algorithm that automatically finds a near-optimal $K$.

Various initialization methods have been investigated to improve the clusters found by KM. Two popular algorithms include Forgy’s method [13], in which the initial centers are assigned as $K$random observations in the data set, and MacQueen’s method (the Random Partition method) [14], in which objects are randomly assigned to clusters and the initial centers are calculated as the centroids of these clusters. However, in a comparative study of initialization methods for K-means, Celebi et al. [15] showed there are alternative initialization methods that outperform these popular algorithms. Additionally, Fränti and Sieranoja [16] found that data sets with well-separated clusters still pose a challenge for all tested existing initialization methods.

Beyond experimenting with initialization methods, researchers have studied the use of different performance functions to improve cluster quality. The Fuzzy K-Means (FKM) algorithm, proposed by Dunn [17], as well as the K-Harmonic Means algorithm, proposed by Zhang et al. [18], use new performance functions that account for soft membership such that every point influences the location of each center. The K-Harmonic Means algorithm applies varying weights to points to further optimize the output, but KHM still converges upon a local minimum rather than the global minimum [18].

Metaheuristics serve as a different method to solve this local optima problem [19]. Some authors [20], [21] used the local search heuristic while some others [22], [23] applied the Tabu search method [24], and some others [25], [26] used genetic algorithms. Nature-based metaheuristic methods have also been explored [27]. Additionally, competitive learning algorithms [28], [29] have been proposed, which use an abundance of initial centers (seeds) that “compete,” “cooperate,” or both to eventually yield a given number of cluster centers; these algorithms have the benefit of automatically determine the number of cluster centers. This paper will specifically focus on the use of the simulated annealing (SA) algorithm [30].

This paper will describe the simulated annealing heuristic and its current applications to the clustering problem in Section 2. Section 3 describes our proposed algorithm. Experimental results that compare the proposed algorithm with other clustering algorithms using the simulated annealing heuristic are detailed in Section 4. Section 5 covers a comparison of center-based, partitional clustering algorithms through generative art.