A reproducing kernel Hilbert space approach to high dimensional partially varying coefficient model

Abstract

Partially varying coefficient model (PVCM) provides a useful class of tools for modeling complex data by incorporating a combination of constant and time-varying covariate effects. One natural question is that how to decide which covariates correspond to constant coefficients and which correspond to time-dependent coefficient functions. To handle this two-type structure selection problem on PVCM, those existing methods are either based on a finite truncation way of coefficient functions, or based on a two-phase procedure to estimate the constant and function parts separately. This paper attempts to provide a complete theoretical characterization for estimation and structure selection issues of PVCM, via proposing two new penalized methods for PVCM within a reproducing kernel Hilbert space (RKHS). The proposed strategy is partially motivated by the so-called “Non-Constant Theorem” of radial kernels, which ensures a unique and unified representation of each candidate component in the hypothesis space. Within a high-dimensional framework, minimax convergence rates for the prediction risk of the first method is established when each unknown time-dependent coefficient can be well approximated within a specified RKHS. On the other hand, under certain regularity conditions, it is shown that the second proposed estimator is able to identify the underlying structure correctly with high probability. Several simulated experiments are implemented to examine the finite sample performance of the proposed methods.

Introduction

Varying coefficient models are a class of useful tools for studying the time-dependent effects of variables. They provide a more flexible approach than the classical linear models and have particular advantages in analyzing dynamic effects of covariates on responses measured repeatedly.

Suppose that ${\left\{(y_i,t_i,{\mathbf{x}}_i)\right\}}_{i=1}^n$ is an independent and identically distributed (i.i.d.) random sample, drawn from the varying coefficient model

$$y=\sum _{j=1}^p{f}_j^o\left(t\right)x_j+\epsilon ,$$

where $\mathbf{x}=(x_1,\dots ,x_p){}^{\mathrm{T}}\in {[-1,1]}^p$ is the $p$-dimensional covariate vector, and each $f_j^o$ is an unknown coefficient function and the noise term $\epsilon$ is standard normal variable. $y\in \mathbb{R}$ is the response of interest and $t\in [0,1]$ is the univariate index variable, typically, $t$ refers to time or age in many applications. We assume that $\mathbf{x}$ and $t$ are independent for our analysis.

The standard varying coefficient models (Fan and Zhang, 1999, Huang et al., 2002, Yang et al., 2006) assume that all time dependent covariates have the same degree of smoothness and are spatially homogeneous, nevertheless, such assumption for the model (1.1) may lead to overfitting and lack of interpretation when some covariate effects are indeed time-invariant, or some of coefficients are irrelevant to the response. This partially motivates us to investigate the PVCM in high dimensional setting. Under the standard structural sparse setting, one often assumes that only $s_0$ out of $p$ functions $f_j^o$’s are non-zero constant and independent of the index $t$, $s_1$ functions are time-varying functions and $(p-s_1-s_0)$ functions are identically equal to zero. Under high dimensional setting, of primary interest is the case where the additive form in (1.1) is fairly sparse, so that $s_0+s_1\ll p$. Indeed, such assumption can be relaxed to general setting where the true functions can be well approximated by the candidate sparse function space. Our main focus here is on situation where $p$ is larger than the sample size $n$.

In this paper, we primarily investigate the estimation of the model coefficients and the identification of a model structure for the model (1.1) under the high-dimensional setting. For this purpose, we present two penalized approaches based on different mixed ${\ell }_1$ penalties within a RKHS. The first proposed method achieves minimax rates in terms of prediction ability and the second one with a simpler penalty ensures structure selection consistency with high probability, which will be stated clearly in Theorem 1, Theorem 2, Theorem 3.

The problem, known variously as model selection, pattern recovery or structure identification, arises in a broad variety of contexts (Donoho, 2006, Miller, 1990, Koller and Friedman, 2009), and such problem in the varying coefficient models has been studied in existing literature. For example, Wang et al. (2007) and Wang and Xia (2008) use the group Lasso and SCAD methods for estimation and model selection in varying coefficient models with a fixed number of coefficients and covariates. In recent years, there have been several existing works for the high dimensional varying coefficient models, and the screening procedure and those penalized methods are used commonly, such as Song et al. (2014), Fan et al. (2014) and Liu et al. (2014), and others. We notice that, these above work treat all the relevant coefficients to be functional forms and do not consider the problem of structure identification including constant coefficients as mentioned above.

Usually, there is no (or few) priori knowledge that which coefficients are indeed constant and which are time-varying functions. In addition to comments as earlier, treating constant coefficients as time-varying would result in a loss in the convergence rate. Hence, identifying multiple structures automatically based on data is an interesting issue for statistical inference. Several other recent papers have considered such multiple structure problems in high dimensional varying coefficient models, for example, Cheng et al. (2014) first adopted a screening procedure to reduce the number of covariates to a moderate order, and then identified both the varying and constant coefficients using a group SCAD-based approach. However, the first step of screening technique is often unable to identify group-relevant features subset while every feature may be very weak to the response. This further affects the quality of model estimation and structure identification. Using the quadratic approximation for local log-likelihood function and the adaptive group Lasso, Li et al. (2015) introduced a penalized weighted least squares procedure to select the relevant covariates and then identify the constant coefficients among the coefficients of the selected covariates. Nevertheless, this method proposed by Li et al. (2015) is an inefficient estimation to some extent, since the first step does not use the information of constant coefficients. Klopp and Pensky (2015) derived the minimax lower bounds within certain frameworks for high dimensional PVCM. In addition, based on a finite basis approximation, an adaptive estimator for the PVCM is constructed with the group Lasso penalty, which achieves those lower bounds under some regularity conditions.

In the current article, we also consider the PVCM when the solution is sparse, and some of relevant coefficients are constant and the rest of relevant coefficients are time dependent. We find that, a key problem for estimating the PVCM (1.1) is the unique representation of each candidate estimator. Otherwise, it is hard to identify constant terms from the derived estimators. Fortunately, Corollary 4.44 in [33] has proved the “no constants” theorem for Gaussian RKHSs, which says that Gaussian RKHSs do not contain any non-zero constant under mild conditions. This conclusion has been extended to general radial kernels (Scovel et al., 2010). Based on this conclusion and considering that those kernel-based approaches are developed in a unified mathematical framework and well-justified in theory, we propose two new penalized approaches to the PVCM within RKHS. To be precise, our first proposed approach is based on a mixed penalty term, that is, the standard ${\ell }_1$-norm penalty for selecting sparse vector coefficients and a combination of an empirical ${\ell }_2$-norm of function and the RKHS-norm for selecting sparse function-type coefficients. The use of the empirical ${\ell }_2$-norm of function in our first method enables us to derive minimax rates under mild conditions, which is inspired partially by several previous work such as Koltchinskii and Yuan, 2010, Raskutti et al., 2012, Suzuki and Sugiyama, 2013 and among others. On the other hand, to ensure structure selection consistency, it suffices to only use the RKHS-norm in our second method.

Compared with existing approaches mentioned as above, our methods are formulated only by one step. Moreover, Our infinite dimensional estimation approach defined on the RKHSs avoids the estimation bias, which is often resulted from the standard finite truncation way of basis functions such as spline functions. The main goal of this paper is to provide a complete theoretical characterization for estimation and structure selection issues of PVCM, via proposing two new penalized methods for PVCM within a RKHS.

Our main contribution of the first proposed method is to establish upper bounds of prediction errors on ${\Vert \cdot \Vert }_2$ and ${\Vert \cdot \Vert }_n$ norms under two different kinds of conditions on covariates, respectively. More precisely, Theorem 1 provides upper bounds on the prediction error under very general conditions, without any “correlation” constraint on covariates. We also show that our derived rates are nearly optimal under certain frameworks. If additional mutual incoherent conditions are satisfied on covariates, like restricted eigenvalue condition (Bickel et al., 2009), we provide sharp upper bounds on the prediction error for the proposed estimator. Moreover, our upper bounds are shown to be optimal in the literature of the PVCM, with appropriate choices of regularization parameters. In addition, our estimator is adaptive to the degree of sparsity.

Our primary contribution of the second proposed method is to prove structure selection consistency in high dimensional setting. Under invertible conditions (Assumption F) and strict incoherent conditions (Assumption G), our second penalized approach achieves model consistency in high dimensional setting. To our knowledge, our results on structure selection consistency via the standard Lasso are new in the framework of high dimensional PVCM. In general, estimation consistency does not guarantee exact recovery of the underlying structure pattern, and studying model consistency requires more stringent conditions such as “irrepresentable” conditions (Zhao and Yu, 2006). In the case of infinite dimensional problems, greater technique challenge is encountered than any analyzing finite dimensional problem. It is also worth noting that, our analysis combines the finite dimensional vector-valued coefficients with the infinite dimensional function-valued coefficients, which is unified in theory and of significant difference from existing work (Cheng et al., 2014, Li et al., 2015).

Finally, our general bounds also provide useful bounds on convergence rates for the two kinds of fundamental statistical models of interest: including linear regression and classical varying-coefficients models. Especially our theoretical results shed light on the connection and differences between these models. Also, the derived conclusion for structure selection consistency can be viewed as an extension of sparse linear models (Wainwright, 2009) and sparse varying-coefficients models.

This work mainly contributes to the statistical theory of parsimonious semiparametric varying-coefficient models by providing both estimation rates and model selection consistency. We also empirically verify its performances through numerical studies. The rest of the paper is organized as follows. In Section 2, we provide background on kernel spaces and some useful properties of RKHS for establishing our penalized methods. Section 3 is devoted to the statement of our main results and discussion of their consequences; it contains the upper bounds on the prediction error under different settings. Structure selection consistency of our second estimator is proved in Section 4, and several simulation results on the first proposed method are shown in Section 5. We present a real data application in Section 6. In Section 7, we provide an explicit proof of Theorem 2, and the other technical details are deferred to the Appendices.

Notation

For a vector $v\in {\mathbb{R}}^m$, ${\Vert v\Vert }_2=\sqrt{{〈v,v〉}_{{\ell }_2}}$ is the standard ${\ell }_2$-norm in the Euclidean space. For a matrix $A$, we define ${\Vert A\Vert }_2={sup}_{{\Vert v\Vert }_2=1}{\Vert Av\Vert }_2$ as the spectral norm of matrix. For a function $f$, define the square-integral norm by ${\Vert f\Vert }_2^2={\int }_{X}f{\left(x\right)}^{2}d\rho \left(x\right)$ associated with some distribution $\rho$. For any vector function $\mathbf{q}\left(t\right)=(q_1\left(t\right),\dots ,q_p\left(t\right)){}^{\mathrm{T}}$, denote ${\Vert \mathbf{q}\Vert }_2=\sqrt{{\sum }_{j=1}^p{\Vert q_j\Vert }_2^2}$. Besides, $a\asymp b$ means that there are two positive constants $c,C$ such that $ca\le b\le Ca$. Also, $a=O\left(b\right)$ means that there is some positive constant $C$ such that $a\le Cb$, and also $a=o\left(b\right)$ means that $a∕b\to 0$. We refer to $C$ with various subscripts as universal positive constants, may differ from line to line. For shorthand, we denote $a\vee b=max\{a,b\}$ and $a\wedge b=min\{a,b\}$. $\left[p\right]=\{1,2,\dots ,p\}$ for ease of notation.