Cross-regression for multi-view feature extraction

doi:10.1016/j.knosys.2020.105997

Knowledge-Based Systems

Volume 200, 20 July 2020, 105997

https://doi.org/10.1016/j.knosys.2020.105997 Get rights and content

Abstract

The traditional multi-view feature extraction (MvFE) method usually seeks a latent common subspace where the samples from different views are maximally correlated. Recently, the regression-based method has become one of the most effective feature extraction methods. However, the existing regression-based methods are only suitable for single-view cases. In this paper, we firstly propose a new MvFE method named as cross-regression for MvFE (CRMvFE). CRMvFE designs a novel cross-regression regularization term to discover the relationship between multiple views in the original space, and simultaneously obtains the low-dimensional projection matrix for each view. Furthermore, inspired by the robustness of L2,1-norm, we also propose a robust CRMvFE (RCRMvFE) and an iterative algorithm to find the optimal solution. Theoretical analysis of the convergence and the relationship with CRMvFE demonstrate the effectiveness of the proposed RCRMvFE. Experiments on datasets show that the proposed CRMvFE and RCRMvFE have better performance than other related methods.

Introduction

With the increase of information collection channels, data is often collected from diverse feature extractors. For example, an image can be represented by different types of papers, and a web page can be represented by hyperlinks and content texts. These different types of samples are called multi-view data. Obviously, multi-view data may characterize different specific information and contains more information than the single-view data. Multi-view learning (MVL) [1], [2], [3], [4], [5], [6] is proposed to integrate compatible and complementary information among different views, which has better performance than traditional single-view learning. Recently, MVL has been widely expanded to many fields, such as multi-view multi-instance learning [7], multi-view clustering [8], [9] and multi-view feature extraction, etc. [10].

In practical applications, the samples usually locate in a high dimensional space, such as image classification [11] or clustering [12]. If we deal with the high-dimensional samples directly, it will lead to the “curse of dimensionality” [13]. So, feature extraction is always a crucial processing step to obtain a tractable low-dimensional representation [14], [15], [16]. Traditional feature extraction methods include Principal Component Analysis (PCA) [17] and Linear Discriminant Analysis (LDA) [18]. However, they are only suitable for single-view data and not for multi-view data. For multi-view data, various multi-view feature extraction (MvFE) methods are proposed, which exploit the correlation information from multiple views when extracting features. Canonical correlation analysis (CCA) [19] and Partial Least Squares (PLS) [20] are two typical methods, which aim at maximizing between-view correlation and covariance respectively. Sun and Wang et al. proposed locality preserving canonical correlation analysis (LPCCA) [21] and a new LPCCA (ALPCCA) [22] respectively. LPCCA and ALPCCA find the low-dimensional embedding by preserving local neighbor information. The main difference is that the former maximizes local canonical correlation coefficient, while the later explores extra cross-view correlations between neighbors. Nevertheless, the number of neighbors in both methods is manually chosen by experience, which affects the final results. Thus, Zu et al. proposed canonical sparse cross-view correlation analysis (CSCCA) [23] which combines sparse reconstruction and LPCCA to explore local intrinsic geometric structure automatically. Zhu et al. pointed out that CSCCA neglects the weights of data, as the difference among samples is not well modeled. Therefore, a weight-based CSCCA (WCSCCA) [24] was proposed. WCSCCA measures the correlation between two views using the weights of data and the cross-view information. For two-view feature extraction, Zhao et al. proposed co-training locality preserving projections (Co-LPP) [25] which aims at finding a low-dimensional embedding such that the local neighbor structures of two views are maximumly compatible.

Note that, all of the above methods are only suitable for two-view scenario. In order to deal with multi-view scenario, Foster et al. proposed Multi-view CCA (MCCA) [26] which tries to find a common space by maximizing the total canonical correlation coefficients between any two views. As a further extension, Cao et al. proposed multi-view PLS (MvPLS) [27]. By unifying Laplacian eigenmaps (LE) [28] and multi-view learning, Xia et al. proposed multi-view spectral embedding (MSE) [29]. MSE finds a subspace in which the low-dimensional embedding is sufficiently smooth. Wang et al. adopted a novel locality linear embedding scheme to develop a new method named multi-view reconstructive preserving embedding (MRPE) [30]. Combining sparse reconstruction and co-regularized scheme, a co-regularized multi-view sparse reconstruction embedding (CMSRE) [31] was proposed. A common characteristic of MSE, MRPE and CMSRE is that they find the low-dimensional embedding directly, and it is unclear how to calculate the low-dimensional presentation of a new point. Therefore, sparsity preserving multiple canonical correlation analysis (SPMCCA) [32] and graph multi-view canonical correlation analysis (GMCCA) [33] were proposed.

Most MvFE methods have two characteristics: (1). They mainly explore the cross-view correlations in the projected subspace without considering the correlation information in original high-dimensional space; (2). They are sensitive to the outliers due to using L2-norm or F-norm [34], [35], [36]. Recently, ridge regression (RR) has made a breakthrough, which uses the previous information (original data or label information) directly in the regression models. There appears many feature extraction methods based on RR for single-view data, such as robust discriminant regression (RDR) [37] and generalized robust regression (GRR) [38]. They aim at finding a robust subspace to maintain the local manifold structure.

Inspired by the regression strategy for feature extraction, this paper constructs a novel cross-regression regularization term to discover the relationship between multiple views in original high-dimensional space. Firstly, minimizing the proposed regularization term aims at seeking a set of projection matrices to transform the samples from different views into a common low-dimensional subspace. Then, another set of projection matrices is introduced to transform the low-dimensional samples back to the original high-dimensional space. Minimizing our proposed cross-regression regularization term is to minimize the distance between original data and the projected high-dimensional data. Finally, two novel MvFE methods, cross-regression for multi-view feature extraction (CRMvFE) and robust cross-regression for multi-view feature extraction (RCRMvFE) are proposed. The main contributions of this paper can be concluded as follows:

(1) A novel cross-regression regularization term is designed and a regression method with this regularization term named CRMvFE is proposed. CRMvFE makes better use of previous information (high-dimensional data) than traditional MvFE methods, and it explores the correlations from multiple views and single-view.

(2) A robust CRMvFE (RCRMvFE) is proposed for extracting robust subspace embedding by adding L2,1-norm instead of F-norm. Furthermore, the influence of outliers can be effectively reduced by using L2,1-norm.

(3) An effective iterative algorithm for solving RCRMvFE is proposed. Theoretical analysis about convergence of the algorithm and the relationship between RCRMvFE and CRMvFE are discussed.

(4) Experimental results on image datasets and hyperspectral image datasets demonstrate the validity and advantage of CRMvFE and RCRMvFE.

The rest of the paper is organized as follows: In Section 2, some related works are introduced. Section 3 presents the proposed CRMvFE. An improved RCRMvFE and its theoretical analysis are presented in Section 4. The experimental results and the conclusion are given in Section 5 and Section 6 respectively.

Section snippets

Related works

In this section, we briefly introduce two typical multi-view feature extraction methods: multi-view canonical correlationanalysis (MCCA) and graph multi-view canonical correlation analysis (GMCCA).

Given the data matrices: $\begin{matrix} X_{A} = {[x_{1}^{A}, x_{2}^{A}, \dots, x_{N}^{A}]}^{T} \in R^{N \times D^{A}}, A \in \{1, \dots, V\} \end{matrix}$ where $X_{A}$ represents the sample matrix from the $A$ th view, $x_{i}^{A}, i = 1, \dots, N$ represents the $i$ th sample of the $A$ th view, $D^{A}$ is the feature dimension in $A$ th view. Given $d < D^{A}, A = 1, \dots, V$ , multi-view feature extraction aims to find the projection matrices $P_{A} \in R^{D^{A} \times}$

Model of CRMvFE

A new MvFE named as cross-regression for multi-view feature extraction (CRMvFE) is proposed in this section. CRMvFE preserves the correlation between multiple views directly by introducing a novel cross-regression regularization term and explores the correlation in single-view itself. For the data matrices given by (1), the sketch of CRMvFE is illustrated in Fig. 1.

CRMvFE finds the projection matrices $P_{A} \in R^{D_{A} \times d}, A = 1, \dots, V$ , by solving the following optimization problem: $\begin{matrix} min_{\binom{P_{B}, F_{A}}{A, B = 1, \dots, V}} & \sum_{A, B = 1}^{V} \sum_{i = 1}^{N} ‖ \end{matrix}$

Robust Cross-regression for Multi-view Feature Extraction (RCRMvFE)

CRMvFE proposed in the above section utilizes F-norm as the metric, which is sensitive to outliers or noises. Therefore, a novel robust CRMvFE (RCRMvFE) is constructed, where L2,1-norm is utilized.

Experiments

In this section, three image datasets and two hyperspectral image datasets are applied to validate the effectiveness of CRMvFE and RCRMvFE. We compare our methods with five well known multi-view feature extraction methods: CoLPP, MCCA, SPMCCA, MvPLS and GMCCA. All the experiments are implemented in Matlab 2016a on a PC with 16RAM and the code of the proposed methods can be downloaded from http://www.scholat.com/zhangjinxin.

Conclusion

In this paper, we propose two novel methods called CRMvFE and RCRMvFE for MvFE. A novel cross-regression regularization term is designed to explore the correlations among multiple views and simultaneously obtain the low-dimensional projection matrices for each view. In addition, in order to reduce the influence of outliers or noises, we propose RCRMvFE which combines CRMvFE and L2,1-norm. Theoretical analysis about convergence of the algorithm and the relationship between CRMvFE and RCRMvFE are

CRediT authorship contribution statement

Jinxin Zhang: Data curation, Formal analysis, Investigation, Software, Methodology, Validation, Visualization, Writing - original draft, Writing - review & editing. Ling Jing: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing - original draft, Writing - review & editing. Junyan Tan: Conceptualization, Data curation, Formal analysis, Investigation, Software, Methodology, Validation,

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Nos. 11671032)

References (38)

ZhouP. et al.
Incremental multi-view spectral clustering
Knowl.-Based Syst.
(2019)
ZhangG. et al.
One-step kernel multi-view subspace clustering
Knowl.-Based Syst.
(2019)
ZhangY. et al.
A multitask multiview clustering algorithm in heterogeneous situations based on LLE and LE
Knowl. Based Syst.
(2019)
WangH. et al.
A study of graph based system for multiview clustering
Knowl. Based Syst.
(2019)
ChenW. et al.
2DRLPP Robust two dimensional locality preserving projection with regularization
Knowl.-Based Syst.
(2019)
DengT. et al.
Low rank local tangent space embedding for subspace clustering
Inform. Sci.
(2020)
LiuK. et al.
An efficient selector for multigranularity attribute reduction
Inform. Sci.
(2019)
SpathisD. et al.
Interactive dimensionality reduction using similarity projections
Knowl.-Based Syst.
(2019)
XieL. et al.
Interactive dimensionality reduction using similarity projections
Knowl.-Based Syst.
(2018)
SunT. et al.
Locality preserving CCA with applications to data visualization and pose estimation
Image Vis. Comput.
(2007)

WangH. et al.

Multi-view reconstructive preserving embedding for dimension reduction

Soft Comput.

(2019)

WangH. et al.

Co-regularized multi-view sparse reconstruction embedding for dimension reduction

Neurocomputing

(2019)

LiuJ. et al.

A joint-L2,1-norm-constraint-based semi-supervised feature extraction for RNA-Seq data analysis

Neurocomputing

(2017)

KangZ. et al.

Multi-graph fusion for multi-view spectral clustering

Knowl.-Based Syst.

(2019)

FengL. et al.

Multi-view locality low-rank embedding for dimension reduction

Knowl.-Based Syst.

(2019)

LaiT. et al.

Efficient robust model fitting for multistructure data using global greedy search

IEEE Trans. Cybern.

(2019)

ZhangJ. et al.

Collaborative weighted multi-view feature extraction

Eng. Appl. Artif. Intell.

(2020)

HeC. et al.

Clustering based multiple instance learning with multiview feature

Expert Syst. Appl.

(2019)

WangX. et al.

Weighted low-rank representation-based dimension reduction for hyperspectral image classification

IEEE Geosci. Remote Sens. Lett.

(2017)

Cited by (15)

Relaxed multi-view discriminant analysis
2024, Engineering Applications of Artificial Intelligence
Consistency and complementarity are two important principles in multiview feature extraction. However, most current multiview feature extraction methods only explore the former but neglect the latter. To alleviate this limitation, in this article we propose a relaxed multiview discriminant analysis (RMDA) model. Firstly, a relaxed loss function is formulated to make the projection matrices have more degrees of freedom. Then two scatter matrices are utilized to preserve cross-view between-class and within-class discriminative information. The proposed RMDA explores the complementarity of multiple views while maintaining the consistency across different views. To solve the RMDA problem efficiently, an iteration strategy is proposed. Theoretical analysis demonstrates the effectiveness and quadratic convergence rate of the RMDA algorithm. To further deal with nonlinearities present in the data, a relaxed kernel multiview discriminant analysis (RKMDA) is put forward too. Several corroborating numerical tests using artificial dataset and real datasets are provided to showcase the merits of the RMDA and RKMDA relative to several competing methods.
Multi-view robust regression for feature extraction
2024, Pattern Recognition
Recently, Multi-view Discriminant Analysis (MVDA) has been proposed and achieves good performance in multi-view recognition tasks. However, as an extension of LDA, this method still suffers from the small-class problem and has the sensitivity to outliers. In order to address these drawbacks and achieve better performance on multi-view recognition tasks, we proposed Multi-view Robust Regression (MVRR) for multi-view feature extraction. MVRR is a regression based method that imposes $L_{2, 1}$ norm as the metric of the loss function and the regularization term to improve robustness and obtain jointly sparse projection matrices for effective feature extraction. Moreover, we incorporate an orthogonal matrix to regress the extracted features to their scaled label to avoid the small-class problem. Therefore, MVRR guarantees the projection matrix to break through the restriction of the number of class for solving the small-class problem. We also propose an iterative algorithm to compute the optimal solution of MVRR and the convergence of MVRR is proved. Experiments are conducted on four databases to verify the performance of MVRR and the result illustrates that MVRR is robust on multi-view feature extraction.
Multi-view clustering via optimal transport algorithm
2023, Knowledge-Based Systems
The surge in data with multiple views has propelled significant interest in the domain of multi-view clustering. Unlike conventional single-view data, multi-view data offers a more accurate representation of objects. However, the pivotal challenge remains in the effective categorization of data through feature extraction from multiple views within clustering tasks. Notably, prevailing multi-view clustering algorithms often emphasize the derivation of appropriate view weights, inadvertently sidestepping optimization intricacies. This approach frequently leads to protracted computational time due to the resource-intensive matrix multiplication operations involved in optimization, coupled with the necessity of weight allocation for diverse views. Addressing this, we propose an optimization framework founded on the optimal transport algorithm that operates independently of view weights. A paramount advantage of the optimal transport algorithm lies in its rapid convergence to a closed-form solution. This study diverges from the conventional focus on view weights and centers on the optimization process within clustering algorithms for resolving multi-view challenges. The introduced framework employing the optimal transport algorithm significantly mitigates computational complexity while handling multiple views. Rigorous experimentation duly substantiates the efficacy of the proposed framework in the realm of multi-view clustering. Across publicly available multi-view datasets, our framework exhibits superior performance over existing state-of-the-art algorithms.
Generalized multiview regression for feature extraction
2023, Information Sciences
Multiview learning (MVL) has attracted considerable attention since an object can be observed from various views. To use the consistency of multiple views, canonical correlation analysis (CCA) is used as a basic technique for analyzing correlated subspaces. However, CCA-based methods are designed to extract the correlation information between each pair of views. In addition, these methods ignore view-specific geometric structures, which may provide effective complementary information. To address the limitations of CCA-based methods and improve the performance of multiview subspace learning methods, a novel MVL framework called generalized multiview regression (GMR) is proposed. It aims at finding a common subspace to preserve the complementary information of each view and maintain consistency among all the views. Specifically, to preserve the view-specific structures, GMR first considers data reconstruction and local geometrical structures. Subsequently, by introducing an orthogonal dictionary, GMR captures the discriminative consistency suitable for classification tasks. Finally, it uses $L_{2, 1}$ as the basic norm to measure errors and regularization, which facilitates robustness and sparsity for feature extraction and selection. An iterative algorithm is designed to solve the proposed GMR. In addition, the convergence and complexity are analyzed theoretically. Extensive experiments on benchmark datasets are conducted to compare the GMR with state-of-the-art or some available multiview methods. The competitive performance implies that GMR is an effective multiview method for learning sparse projections and extracting discriminative and robust features.
Multiview Jointly Sparse Discriminant Common Subspace Learning
2023, Pattern Recognition
Multiview data leads to the demand for classifying samples from various views, and the large gap between different views makes the classification task challenging. Recently, researchers have extended linear discriminant analysis (LDA) to multi-view scenarios. However, the extended methods are generally associated with the small-class problem, that is, the projection size is limited by the number of classes. In addition, they are sensitive to variations in images or outliers. To solve these problems, this study proposes a generalized robust multiview discriminant analysis (GRMDA) to obtain a linear transform for each view and for learning multiview jointly sparse discriminant common subspace. GRMDA aims to achieve both maximal between-class and minimal within-class variation for data from multiple views in a common space. Instead of formulating the ratio trace problem, we reformulate GRMDA inspired by maximum margin criterion (MMC) to address the small-class problem. Moreover, the proposed method achieves stronger robustness by reconstructing the within-class and between-class scatter terms from the definition of $L_{2, 1}$ norm. Furthermore, GRMDA ensures joint sparsity using the $L_{2, 1}$ norm-based regularization term. Additionally, we present an iterative algorithm, convergence proof, and complexity analysis. Experiments on six popular databases, that is, COIL100, USPS/MNIST, Extended Yale Face B, AR, BBCSport, and multiple feature datasets, were conducted to evaluate the performance of GRMDA against the state-of-the-art multiview methods. The experimental results demonstrate that the proposed method can achieve a significant performance with strong robustness and fast convergence.
Orthogonal multi-view analysis by successive approximations via eigenvectors
2022, Neurocomputing
Citation Excerpt :
To take full advantage of multi-view data, multi-view learning has attracted increasing attention due to its wide applications such as dimensionality reduction [1], cross-view recognition [2,3], clustering [4,5], classification [6], and multi-label learning [7,8]. Many learning criteria have been explored to capture the relations among multiple views including subspace learning methods [9,10], tensor approaches [11,12] and the deep learning [13–15]. Although great progress has been made by existing multi-view learning methods, there are still challenges.
Orthogonality has been demonstrated to admit many desirable properties such as noise-tolerant, good for data visualization, and preserving distances. However, it is often incompatible with existing models and the resulting optimization problem is challenging even if compatible. To address these issues, we propose a trace ratio formulation for multi-view subspace learning to learn individual orthogonal projections for all views. The proposed formulation integrates the correlations within multiple views, supervised discriminant capacity, and distance preservation in a concise and compact way. It not only includes several existing models as special cases, but also inspires new models. Moreover, an efficient numerical method based on successive approximations via eigenvectors is presented to solve the associated optimization problem. The method is built upon an iterative Krylov subspace method which can easily scale up for high-dimensional datasets. Extensive experiments are conducted on various real-world datasets for multi-view discriminant analysis and multi-view multi-label classification. The experimental results demonstrate that the proposed models are consistently competitive to and often better than the compared methods.

View all citing articles on Scopus

View full text

Cross-regression for multi-view feature extraction

Abstract

Introduction

Section snippets

Related works

Model of CRMvFE

Robust Cross-regression for Multi-view Feature Extraction (RCRMvFE)

Experiments

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgments

Knowl.-Based Syst.

Knowl.-Based Syst.

Knowl. Based Syst.

Knowl. Based Syst.

Knowl.-Based Syst.

Inform. Sci.

Inform. Sci.

Knowl.-Based Syst.

Knowl.-Based Syst.

Image Vis. Comput.

Soft Comput.

Neurocomputing

Neurocomputing

Multi-graph fusion for multi-view spectral clustering

Knowl.-Based Syst.

Multi-view locality low-rank embedding for dimension reduction

Knowl.-Based Syst.

Efficient robust model fitting for multistructure data using global greedy search

IEEE Trans. Cybern.

Collaborative weighted multi-view feature extraction

Eng. Appl. Artif. Intell.

Clustering based multiple instance learning with multiview feature

Expert Syst. Appl.

Weighted low-rank representation-based dimension reduction for hyperspectral image classification

IEEE Geosci. Remote Sens. Lett.