Structured graph learning for clustering and semi-supervised classification

Highlights

• A graph learning framework, which captures both the global and local structure in data, is proposed.

• Theoretical analysis builds the connections of our model to k-means, spectral clustering, and kernel k-means.

• Extensions to semi-supervised classification and multiple kernel learning are presented.

Abstract

Graphs have become increasingly popular in modeling structures and interactions in a wide variety of problems during the last decade. Graph-based clustering and semi-supervised classification techniques have shown impressive performance. This paper proposes a graph learning framework to preserve both the local and global structure of data. Specifically, our method uses the self-expressiveness of samples to capture the global structure and adaptive neighbor approach to respect the local structure. Furthermore, most existing graph-based methods conduct clustering and semi-supervised classification on the graph learned from the original data matrix, which doesn’t have explicit cluster structure, thus they might not achieve the optimal performance. By considering rank constraint, the achieved graph will have exactly c connected components if there are c clusters or classes. As a byproduct of this, graph learning and label inference are jointly and iteratively implemented in a principled way. Theoretically, we show that our model is equivalent to a combination of kernel k-means and k-means methods under certain condition. Extensive experiments on clustering and semi-supervised classification demonstrate that the proposed method outperforms other state-of-the-art methods.

Introduction

As a natural way to represent structure or connections in data, graphs have broad applications including world wide web, social networks, information retrieval, bioinformatics, computer vision, natural language processing, and many others. Some special cases of graph algorithms, such as graph-based clustering [1], [2], graph embedding [3], graph-based semi-supervised classification [4], signal processing [5], have attracted increasing attention in the recent years.

Clustering refers to the task of finding subsets of similar samples and grouping them together, such that samples in the same cluster would share high similarity to each other, whereas samples in different groups are dissimilar [6], [7]. By leveraging a small set of labeled data, semi-supervised classification aims at determining the labels of a large collection of unlabeled samples based on relationships among the samples [8]. In essence, both clustering and semi-supervised classification algorithms are trying to predict labels for samples [9]. As fundamental techniques in machine learning and pattern recognition, they have been facilitating various research fields and have been extensively studied.

Among numerous clustering and semi-supervised classification methods developed in the past decades, graph based techniques often provide impressive performance. In general, these methods consist of two key steps. First, an affinity graph is constructed from all data points to represent the similarity among the samples. Second, spectral clustering [10] algorithm or label propagation [11] method is utilized to obtain the final labels. Therefore, the start step of building graph might heavily impact the subsequent step and finally lead to suboptimal performance. Since underlying structures of data are often unknown in advance, this pose a major challenge for graph construction. Consequently, the final result might be far from optimal. Unfortunately, constructing a good graph that best captures the essential data structure is still known to be fundamentally challenging [12].

The existing strategies to define adjacency graph can be roughly divided into three categories: a) the metric based approaches, which use some functions to measure the similarity among data points [13], such as Cosine, Euclidean distance, Gaussian function; b) the local structure approaches, which induce the similarity by representing each datum as a linear combination of local neighbors [14] or learning a probability value for two points as neighbors [15]; c) the global self-expressiveness property based approaches, which encode each datum as a weighted combination of all other samples, i.e., its direct neighbors and reachable indirect neighbors [16], [17]. The traditional metric based approaches and the local neighbor based methods depend upon the selection of metric or the local neighborhood parameter, which heavily influence final accuracy. Hence, they are not reliable in practice [18].

On the other hand, adaptive neighbor [15] and self-expressiveness approaches [19], [20] automatically learn graph from data. As a matter of fact, they share a similar spirit as locality preserve projection (LPP) and locally linear embedding (LLE), respectively. Different from LPP and LLE, they don’t specify the neighborhood size and predefine the similarity graph. In realistic applications, they enjoy several benefits. First, automatically determining the most informative neighbors for each data point will avoid the inconsistent drawback in widely used k-nearest-neighborhood and ϵ-nearest-neighborhood graph construction techniques, which provide unstable performance with respect to different k or ϵ values [21]. Second, they are independent of measure metric, while traditional methods are often data-dependent and sensitive to noise and outliers [22]. Third, they can tackle data with structures at different scales of size and density [23]. Therefore, they are prefered in practice. For example, [24] performs dimension reduction and graph learning based on adaptive neighbor in a unified framework.

Nevertheless, they emphasize different aspects of data structure information, i.e., local and global, respectively. As demonstrated in many problems, such as dimension reduction [25], feature selection [26], semi-supervised classification [27], clustering [14], local and global structure information are both important to algorithm performance since they can provide complementary information to each other and thus enhance the performance. In the paper, we combine them into a unified framework for graph learning task.

Moreover, most existing graph-based methods conduct clustering and semi-supervised classification on the graph learned from the original data matrix, which doesn’t have explicit cluster structure, thus they might not achieve the optimal performance. For example, the seminal work [20] assumes a low-rank structure of graph, whose solution might not be optimal due to the bias of nuclear norm [28]. Ideally, the achieved graph should have exactly c connected components if there are c clusters or classes. Most existing methods fail to take this information into account. In this paper, we consider rank constraint to meet this requirement. As an extension to our previous work [22], we establish the theoretical connection of our clustering model to kernel k-means and k-means and consider semi-supervised classification application. As an added bonus, graph learning and label inference are seamlessly integrated into a unified objective function. This is quite different from traditional ways, where graph learning and label inference are performed in two separate steps, which easily lead to suboptimal results. To overcome the limitation of single kernel method, we further extend our model to accommodate multiple kernels.

Though there are many other lines of research on graph. For instance, [29] discusses the transformation issue; [30] introduces a fitness metric to learn the adjacency matrix; [31] focuses on the graph that is sampled from a graphon. Different from them, this work aims to learn a graph that has explicit cluster structure. In particular, the number of clusters/classes is employed as a prior knowledge to enhance the quality of graph, which leads to improved performance of clustering and semi-supervised classification. Additionally, graph neural networks (GNN) has gained increasing popularity recently [32]. The main difference between GNN and our method is that GNN targets to process a graph that is already available in existing data, while our method is designed to learn a good graph from feature data for further processing. Hence, our method and GNN focus on different types of data. In practice, feature data is more common than graph data. From this point of view, our method could be useful for GNN applications when the graph is not available or the graph has low quality. As a matter of fact, how to refine the graph used in GNN is a promissing research direction.

To sum up, the main contributions of this paper are:

• The similarity graph and labels are adaptively learned from the data by preserving both global and local structure information. By leveraging the interactions among them, they are mutually reinforced towards an overall optimal solution.

• Theoretical analysis shows the connections of our model to kernel k-means, k-means, and spectral clustering methods. Our framework is more general than k-means and kernel k-means. At the same time, it solves the graph construction challenge of spectral clustering.

• Based on our method with a single kernel, we further extend our model into an integrated framework which can simultaneously learn the similarity graph, labels, and the optimal combination of multiple kernels. Each subtask can be iteratively boosted by using the results of the others.

• Extensive experiments on real-world data sets are conducted to testify the effectiveness and advantages of our framework over other state-of-the-art clustering and semi-supervised classification algorithms.

The rest of the paper is organized as follows. Section 2 introduces the proposed clustering method based on a single kernel. In Section 3, we show the theoretical analysis of our model. An extended model with multiple kernel learning ability is provided in Section 4. Clustering and semi-supervised classification experimental results and analysis are presented in Section 5 and 6, respectively. Section 7 draws conclusions.

Notations. Given a data set $X\in {\mathcal{R}}^{n\times m}$with m features and n instances, its ith sample and (i, j)th element are denoted by $x_i\in {\mathcal{R}}^{m\times 1}$and x_ij, respectively. The ℓ₂-norm of x_i is denoted as $\parallel x_i\parallel =\sqrt{x_i^T·x_i},$where T means transpose. The definition of squared Frobenius norm is ${\parallel X\parallel }_F^2=\sum _{ij}{x}_{ij}^2$. Irepresents the identity matrix and 1 denotes a column vector with all the elements as one. $Tr(\dot{)}$is the trace operator. 0 ≤ Z ≤ 1 indicates that elements of Z are in the range of [0,1].