Nonparametric variable selection and its application to additive models

Abstract

Variable selection for multivariate nonparametric regression models usually involves parameterized approximation for nonparametric functions in the objective function. However, this parameterized approximation often increases the number of parameters significantly, leading to the “curse of dimensionality” and inaccurate estimation. In this paper, we propose a novel and easily implemented approach to do variable selection in nonparametric models without parameterized approximation, enabling selection consistency to be achieved. The proposed method is applied to do variable selection for additive models. A two-stage procedure with selection and adaptive estimation is proposed, and the properties of this method are investigated. This two-stage algorithm is adaptive to the smoothness of the underlying components, and the estimation consistency can reach a parametric rate if the underlying model is really parametric. Simulation studies are conducted to examine the performance of the proposed method. Furthermore, a real data example is analyzed for illustration.

Appendix

Proof of Theorem 1

Recall the definition of ${\eta }$, $\mathbf {Z},$ and ${\mathbf{A}_d }^{T}\mathbf {X}={\eta }^{T}\mathbf {Z}$. ${A}_1$ is a p-dimensional vector whose first d elements are 1, otherwise 0. We have

$$\begin{aligned} \varvec{\Sigma }^{-1}\mathrm {E}(\mathbf {X} h(Y))= & {} \varvec{\Sigma }^{-1/2}({\mathbf{B}}_1,\eta _1/\Vert \eta _1\Vert )({\mathbf{B}}_1, \eta _1/\Vert \eta _1\Vert )^{\top } \mathrm {E}\left( \mathbf {Z} h(Y)\right) \nonumber \\= & {} \varvec{\Sigma }^{-1/2}{\mathbf{B}}_1{\mathbf{B}}_1^{\top }\mathrm {E}\left( \mathbf {Z} h(Y)\right) + \varvec{\Sigma }^{-1/2}\eta _1 \eta _1^{\top } \mathrm {E}\left( \mathbf {Z} h(Y)\right) /\Vert \eta _1\Vert ^2\nonumber \\= & {} \varvec{\Sigma }^{-1/2}{\mathbf{B}}_1{\mathbf{B}}_1^{\top }\mathrm {E}\left( \mathbf {Z} h(Y)\right) +{A}_1 {A}_1^{\top } \mathrm {E}\left( \mathbf {X} h(Y)\right) /\Vert \eta _1\Vert ^2\nonumber \\=: & {} \varvec{\Sigma }^{-1/2}{\mathbf{B}}_1\mathrm {E}\left( \mathrm {E} ({\mathbf{B}}_1^{\top }\mathbf {Z}|Y) h(Y)\right) +c_h{ A}_1. \end{aligned}$$

(24)

It is obvious that the first term is equal to zero when the condition $\mathrm {E}({\mathbf{B}}_1^{\top }\mathbf {Z}|Y)=0$ almost surely. Thus, (3) implies (4). On the other hand, when (4) holds, the first term in (24) is zero. For any transformation $h(\cdot )$, $\varvec{\Sigma }^{-1/2}{\mathbf{B}}_1\mathrm {E}\big (\mathrm {E} ({\mathbf{B}}_1^{\top }\mathbf {Z}|Y) h(Y)\big ) =0$, implies $\mathrm {E} ({\mathbf{B}}_1^{\top }\mathbf {Z}|Y) =0$ almost surely. (4) implies (3). When the distribution of $\mathbf {Z}$ is elliptically symmetric, the Eq. (4) can be proved similarly in Li and Duan (1989). $\square$

Proof of Theorem 3

Here, we give the sketch of proof of Theorem 3, which is Theorem 1 in Lin et al. (2009). For details please refer to Lin et al. (2009).

To proof Theorem 3, two lemmas are needed.

Lemma 1

Suppose the conditions of Theorem 3 hold and denote

$$\begin{aligned} \hat{f}_{1M}(x_1) = f_{1M}(x_1) \frac{\sum _{i=1}^n \{Y_i - \mu - \sum _{j=2}^d {f}_{jM}(x_{ij})\}\int _{s_{i-1}}^{s_i}K\left( \frac{t-x_1}{h}\right) f_{1M}(t)\mathrm{d}t}{\int _0^1 K\left( \frac{t-x_1}{h}\right) (f_{1M}(t))^2\mathrm{d}t}, \end{aligned}$$

where $f_{1M}(x_j)$ are defined in Sect. 3.1. $s_i, i=0,\ldots ,n$ are defined as $s_0=0$, $s_i=(x_{i1}+x_{(i+1)1})/2, i=1,\ldots ,n-1$, $s_n=1$, $0\le x_{11}<x_{21}<\cdots <x_{n1}\le 1$ ordered. Then, as $h\rightarrow 0$ and $n\rightarrow \infty$, the bias and variance of $\hat{f}_{1M}(x_1)$ can be expressed, respectively, as

$$\begin{aligned} \hbox {bias}(\hat{f}_{1M}(x_1))= & {} \frac{1}{2}h^2 \sigma _K^2M^{-\gamma _{12}}e_{21}(x_1)+ O(n^{-1})+ o(h^2M^{-\gamma _{12}}) + O(M^{-\gamma _{0}}),\\ \hbox {var}(\hat{f}_{1M}(x_1))= & {} \frac{\sigma ^2J_K}{nhp_1(x_1)}+ O(n^{-1})+O(n^{-2}h^{-2}), \end{aligned}$$

where $\gamma _0 = \min \{\gamma _{j0}; j=1,2,\ldots ,d\}$, $e_{21}(x_1)$ is defined in condition C2, and satisfying $\lim _{M\rightarrow \infty }M^{\gamma _{j2}}r_{jM}''(x_j)=e_{j2}(x_j),j=1,\ldots ,d$.

Lemma 2

Let the conditions of Theorem 3 hold and let

$$\begin{aligned} \check{f}_{1M}(x_1)= & {} f_{1M}(x_1) \\&+ \frac{\sum _{i=1}^n \left\{ Y_i - \mu - \sum _{j=2}^d {f}_{jM}(x_{ij})\right\} \int _{s_{i-1}}^{s_i}K\left( \frac{t-x_1}{h}\right) \mathrm{d}t-\int _0^1K \left( \frac{t-x_1}{h}\right) f_{1M}(t)\mathrm{d}t}{\int _0^1 K\left( \frac{t-x_1}{h}\right) }. \end{aligned}$$

Then as $h\rightarrow 0$ and $n\rightarrow \infty$, the bias and variance of $\check{f}_{1M}(x_1)$ can be expressed, respectively, as

$$\begin{aligned} \hbox {bias}(\check{f}_{1M}(x_1))= & {} \frac{1}{2}h^2 \sigma _K^2M^{-\gamma _{12}}e_{21}(x_1)+ O(n^{-1})+ o(h^2M^{-\gamma _{12}}) + O(M^{-\gamma _{0}}),\\ \hbox {var}(\check{f}_{1M}(x_1))= & {} \frac{\sigma ^2J_K}{nhp_1(x_1)}+ O(n^{-1})+O(n^{-2}h^{-2}), \end{aligned}$$

where $\gamma _0 = \min \{\gamma _{j0}; j=1,2,\ldots ,d\}$, $e_{21}(x_1)$ is defined in condition C2, and satisfying $\lim _{M\rightarrow \infty }M^{\gamma _{j2}}r_{jM}''(x_j)=e_{j2}(x_j),j=1,\ldots ,d$.

To proof Theorem  3, in Sect. 3.2, we defined that $r_{jM}(x_j)=f_j(x_j)-f_{jM}(x_j)$; here, we define that

$$\begin{aligned} R_M(x)=\sum _{j=1}^d f_j(x_j)-\sum _{j=1}^d \sum _{l=1}^M \beta ^0_{jl}q_l(x_j). \end{aligned}$$

Then, the first stage estimators of $f_j(x_j)$ can be expressed as

$$\begin{aligned} \tilde{f}_{j}(x_j) = f_{jM}(x_j) + \frac{1}{n}\sum _{i=1}^n PA(x_i)(\varepsilon _i + R_M(x_i)), j= 1,\ldots ,d, \end{aligned}$$

where $PA(x_i)$ is the summation components in equation (A1) in Lin et al. (2009). And using Taylor expansion, $\hat{f}_1(x_1)-\hat{f}_{1M}(x_1)$ can be expanded at $f_{1M}(x_1)$; then, we can get

$$\begin{aligned} \hat{f}_1(x_1)-\hat{f}_{1M}(x_1)= & {} B_1(x_1)(\tilde{f}_1(x_1)-f_{1M}(x_1))\\&+ B_2(x_1)\left\{ \tilde{\mu }-\mu _0 +\sum _{j=2}^d(\tilde{f}_j(x_j)-f_{jM}(x_j))\right\} + o_p(hn^{-1}M), \end{aligned}$$

where $B_1(x_1)=\eta _1/\eta _2+f_1(x_1)\eta _3/\eta 2-2f_1(x_1)\eta _1/\eta _4^2$, and $B_2(x_1)=-f_1(x_1)\eta _5/\eta _2$ with $\eta _1= \sum _{i=1}^n\{Y_i-\mu ^0-\sum _{j=2}^d f_{jM}(x_{ij})\}\int _{s_{i-1}}^{s_i}K(\frac{t-x_1}{h})f_{1M}(t)\mathrm{d}t$, $\eta _2 = \int _0^1 K(\frac{t-x_1}{h})f^2_{1M}(t)\mathrm{d}t$, $\eta _3=\sum _{i=1}^n\{Y_i-\mu ^0-\sum _{j=2}^d f_{jM}(x_{ij})\}\int _{s_{i-1}}^{s_i}K(\frac{t-x_1}{h})\mathrm{d}t$, $\eta _4 = \int _0^1 K(\frac{t-x_1}{h})f_{1M}(t)\mathrm{d}t$, $\eta _5=\sum _{i=1}^n Y_i\int _{s_{i-1}}^{s_i}K(\frac{t-x_1}{h})f_{1M}(t)\mathrm{d}t$.

From the results above, we have

$$\begin{aligned} E\{\eta _1(\tilde{f}_1(x_1)-f_{1M}(x_1))\} = O(hM^{-\gamma _0+1})+O(hn^{-1}M). \end{aligned}$$

And similarly $E\{\eta _3(\tilde{f}_1(x_1)-f_{1M}(x_1))\} = O(hM^{-\gamma _0+1})+O(hn^{-1}M)$. So,

$$\begin{aligned} E\{B_1(x_1)(\tilde{f}_1(x_1)-f_{1M}(x_1))\}=O(M^{-\gamma _0+1})+O(n^{-1}M). \end{aligned}$$

And $E\{B_2(x_1)[\tilde{\mu }-\mu +\sum _{j=2}^d(\tilde{f}_j(x_j)-f_{jM}(x_j))]\} =O(M^{-\gamma _0+1})+O(n^{-1}M)$. Combining these results with Lemma 1 and conditions C1 and C2 leads to the final expression of $\hbox {bias}(\hat{f}_{1}(x_1))$ and $\hbox {var}(\hat{f}_{1}(x_1))$ in Theorem 3.

The second part of the proof is similar, and so we omitted it here. $\square$