Promote sign consistency in the joint estimation of precision matrices

Abstract

The Gaussian graphical model is a popular tool for inferring the relationships among random variables, where the precision matrix provides a natural interpretation of conditional independence. With high-dimensional data, sparsity of the precision matrix is often assumed, and various regularization methods have been applied for estimation. In several scenarios, it is desirable to conduct the joint estimation of multiple precision matrices. In joint estimation, entries corresponding to the same element of multiple precision matrices form a group, and group regularization methods have been applied for the estimation and identification of sparsity structures. In many practical examples, it can be difficult to interpret the results when parameters within the same group have conflicting signs. Unfortunately, existing methods lack an explicit mechanism in regards to sign consistency of group parameters. To tackle this problem, a novel regularization method is developed for the joint estimation of multiple precision matrices. It effectively enhances the sign consistency of group parameters and hence can lead to more interpretable results, while still allowing for conflicting signs to achieve full flexibility. The method’s consistency properties are rigorously established. Simulations show that the proposed method outperforms competing alternatives under a variety of settings. For the two data examples, the proposed approach leads to interpretable results that are different from the alternatives.

Introduction

The Gaussian graphical model has emerged as a popular tool for studying the dependence among random variables (Friedman, 2004, Wainwright and Jordan, 2008). In the Gaussian graphical model analysis, the precision matrix plays a pivotal role: two variables are conditionally independent given all other variables if and only if the corresponding element in the precision matrix is zero. When the data dimensionality is high, sparsity of the precision matrix is often assumed, and identification of nonzero elements (edges) needs to be conducted along with estimation. For this identification, regularization is needed, with the most popular technique perhaps being lasso (Bickel and Elizaveta, 2008, Friedman et al., 2008). To reduce bias caused by the ${\ell }_1$ penalty, Lam and Fan (2009) apply the nonconvex SCAD penalty. Other existing methods include the neighborhood lasso selection (Meinshausen and Bühlmann, 2006), neighborhood Dantzig selection (Yuan, 2010), alternating linearization method (Scheinberg et al., 2010), penalized D-trace loss method (Zhang and Zou, 2014), constrained ${\ell }_1$ minimization (Cai et al., 2011), and others. We refer to Fan et al. (2016) for a comprehensive overview of sparse estimators.

The aforementioned studies as well as many others in the literature are mainly concerned with analyzing a single dataset. However, in many important scenarios, it is desirable to conduct joint estimation with multiple independent datasets. Below we present two examples in genetics, which match the data analyzed in Section 4, and note that similar examples arise in many other fields.

• Example 1. In genetic data analysis, it has been recognized that the results generated from a single dataset are often unsatisfactory (Zhao et al., 2015). For many important problems, there have been multiple independent datasets generated with similar designs. Pooling and jointly analyzing these datasets can effectively increase sample size and estimation/identification accuracy (Ma et al., 2011).

• Example 2. Different diseases can share common molecular mechanisms. To identify these commonalities, it is essential to jointly analyze multiple datasets. This kind of analysis has been conducted by Ma et al. (2012), Liu et al. (2013), and others.

The idea of integrating multiple data sources and learning parameter homogeneity/heterogeneity goes beyond the two examples above and has been studied extensively using the penalized methods, for example by Tang and Song (2016), Cheng et al. (2015), Tang et al. (2020), and others. A popular way to integrate information from multiple datasets is meta-analysis. Some meta-analysis approaches operate on summary statistics, while the individual participant data meta-analysis approaches directly analyze individual level data (Riley et al., 2010). The random-effects model has been proposed to handle heterogeneity in magnitudes, but so far there has been a lack of work on promoting sign consistency. Although some individual participant data meta-analysis approaches can handle high-dimensional data, our literature review suggests that the penalization technique has not been commonly adopted. For example, Waldron et al. (2014) apply fixed-effects and random-effects models on the C-index, and Emura et al. (2018) propose to use a weighted sum of gene expressions for dimension reduction. We note that although the object of estimation in the aforementioned studies is not a precision matrix, the settings and considerations are expected to be similar for precision matrices.

In recent studies, the joint estimation of multiple precision matrices has been investigated. One framework is to incorporate a target matrix such that the estimation can be encouraged to shrink to this target matrix (Bilgrau et al., 2020), or update the precision matrix repeatedly over time with prior information available (van Wieringen et al., 2020). Another framework, which is the framework we adopt, is to jointly analyze multiple raw datasets without any accessible target matrix or prior information. Building on the maximum penalized likelihood, Chiquet et al. (2011) assume a common structure and encourage estimation of multiple networks towards this common structure. Guo et al. (2011) propose a hierarchical penalization approach to jointly estimate multiple graphical models. Danaher et al. (2014) examine the joint graphical lasso using group lasso and fused lasso penalties. Along this line, Saegusa and Shojaie (2016) take a Laplacian penalization approach. Zhu et al. (2014) develop a nonconvex method to search for a sparse representation for each matrix and identify clusters of entries across matrices. Different from the penalized likelihood approaches, Cai et al. (2016) develop a weighted constrained ${\ell }_{\infty }∕{\ell }_1$ minimization approach. With all these methods, a popular strategy is to treat the elements corresponding to the same edge in multiple datasets as a group, and regularization is imposed on the (norms of) group parameters. Despite considerable successes, a common limitation shared by most of the existing studies is a lack of consideration of the relationships across datasets (matrices). In data analysis, it is often sensible to expect that the same edges have the same signs in different datasets although the magnitudes can be different.

In this study, we conduct the joint estimation of multiple precision matrices under high-dimensional settings. Motivated by the above discussions, our goal is to promote sign consistency across precision matrices while flexibly allowing for conflicting signs. The rest of the article is organized as follows. In Section 2, we develop a new joint estimation approach to promote sign consistency. An effective computational algorithm is developed, and the theoretical properties are rigorously established. In Section 3, we conduct extensive simulations and compare our approach with the competing alternatives. Two data examples are analyzed in Section 4. The article concludes with discussions in Section 5. Proofs and additional numerical results are presented in the Supplementary File.