Pairwise Fusion Approach Incorporating Prior Constraint Information

Abstract

In this paper, we explore sparsity and homogeneity of regression coefficients incorporating prior constraint information. The sparsity means that a small fraction of regression coefficients is nonzero, and the homogeneity means that regression coefficients are grouped and have exactly the same value in each group. A general pairwise fusion approach is proposed to deal with the sparsity and homogeneity detection when combining prior convex constraints. We develop a modified alternating direction method of multipliers algorithm to obtain the estimators and demonstrate its convergence. The efficiency of both sparsity and homogeneity detection can be improved by combining the prior information. Our proposed method is further illustrated by simulation studies and analysis of an ozone dataset.

Appendix

1.1 A. Proof of Proposition 2.1

By the definition $\varvec{\eta }_1^{(m+1)}, \mathbf {\varvec{\eta }}_2^{(m+1)}$, for any $\varvec{\eta }_1$, $\mathbf {\varvec{\eta } }_2$, we have

$$\begin{aligned} S_n \biggl (\varvec{\beta }^{(m+1)}, \varvec{\eta }_1^{(m+1)}, \varvec{\eta }_2^{(m+1)}, {\mathbf {v}}^{(m)}_1, {\mathbf {v}}_2^{(m)}\biggr )\le S_n \biggl (\varvec{\beta }^{(m+1)},\varvec{\eta }_1, \varvec{\eta }_2, {\mathbf {v}}^{(m)}_1, {\mathbf {v}}_2^{(m)} \biggr ). \end{aligned}$$

Denote $\Xi (\varvec{\beta }) = \{((\varvec{\eta }_1, \varvec{\eta }_2) : \varvec{\beta }^{(m+1)}- \varvec{\eta }_1 ={\mathbf {0}}, {\mathbf {L}}\varvec{\beta }^{(m+1)}- \varvec{\eta }_2 ={\mathbf {0}} \}$,

$$\begin{aligned} h^{(m+1)}= & {} \inf \limits _{\Xi (\varvec{\beta })} \biggl \{\frac{1}{2n}\biggl \Vert {\mathbf {y}}- {\mathbf {X}}\varvec{\beta }^{(m+1)}\biggr \Vert ^{2}+ {\mathbb {I}}_{{\mathcal {C}}}(\varvec{\eta }_1 ) + p_{\gamma }(|\varvec{\beta }^{(m+1)}|, \lambda _1 ) + p_{\gamma }(|\varvec{\eta }_2|,\lambda _2 ) \biggr \} \\= & {} \inf \limits _{\Xi (\varvec{\beta })}S_n( \varvec{\beta }^{(m+1)}, \varvec{\eta }_1, \varvec{\eta }_2, {\mathbf {v}}_1^{(m)},{\mathbf {v}}_2^{(m)} ). \end{aligned}$$

Then,

$$\begin{aligned} S_n \biggl (\varvec{\beta }^{(m+1)}, \varvec{\eta }_1^{(m+1)}, \varvec{\eta }^{(m+1)}_2, {\mathbf {v}}_1^{(m)}, {\mathbf {v}}_2^{(m)} \biggr )\le h^{(m+1)}. \end{aligned}$$

For any integer t, ${\mathbf {v}}_1^{(m+t-1)} = {\mathbf {v}}_1^{(m)} + \tau _1\sum _{i=1}^{t-1}(\varvec{\beta }_1^{(m+i)} - \varvec{\eta }_1^{(m+i)} )$, and ${\mathbf {v}}^{(m+t-1)}_2={\mathbf {v}}^{(m)}_2+\tau _2 \sum \nolimits _{i=1}^{t-1}({\mathbf {L}}\varvec{\beta }^{(m+i)}- \varvec{\eta }^{(m+i)}_2)$, then we have

$$\begin{aligned}&S_n\biggl (\varvec{\beta }^{(m+t)}, \varvec{\eta }^{(m+t)}_1, \varvec{\eta }_2^{(m+t)}, {\mathbf {v}}^{(m+t-1)}_1,{\mathbf {v}}_2^{(m+t-1)} \biggr ) \\&\quad =\frac{1}{2n}\left\| {\varvec{y}}-{\mathbf {X}}\varvec{\beta }^{(m+t)}\right\| ^{2}+ p_{\gamma }(|\varvec{\beta }^{(m+1)}|, \lambda _1 ) + {\mathbb {I}}_{{\mathcal {C}}}(\varvec{\eta }_1^{(m+t)})\\&\qquad + {\mathbf {v}}_1^{(m+t-1)\mathrm {T}}(\varvec{\beta }^{(m+t)} -\varvec{\eta }_1^{(m+t)}) \\&\qquad +\, \frac{\tau _1}{2}\Vert \varvec{\beta }^{(m+t)} -\varvec{\eta }_1^{(m+t)} \Vert + {\mathbf {v}}^{(m+t-1)\mathrm {T}}_2({\mathbf {L}}\varvec{\beta }^{(m+t)} - \varvec{\eta }^{(m+t)}_2) \\&\qquad +\,\frac{\tau _2 }{2}||{\mathbf {L}}\varvec{\beta }^{(m+t)} - \varvec{\eta }^{(m+t)}_2||^{2} + p_{\gamma }(|\varvec{\eta }_2^{(m+t)}|,\lambda _2 ) \\&\quad =\frac{1}{2n}\left\| {\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{(m+t)}\right\| ^{2}+ p_{\gamma }(|\varvec{\beta }^{(m+1)}|, \lambda _1 ) + {\mathbb {I}}_{{\mathcal {C}}}(\varvec{\eta }_1^{(m+t)})\\&\qquad + {\mathbf {v}}_1^{(m)\mathrm {T}}(\varvec{\beta }^{(m+t)} -\varvec{\eta }_1^{(m+t)})\\&\qquad +\, \tau _1\sum _{i=1}^{t-1}(\varvec{\beta }^{(m+i)} -\varvec{\eta }_1^{(m+i)})^{\mathrm {T}}(\varvec{\beta }^{(m+t)} -\varvec{\eta }_1^{(m+t)}) + {\mathbf {v}}_2^{(m)\mathrm {T}}({\mathbf {L}}\varvec{\beta }^{(m+t)} -\varvec{\eta }_1^{(m+t)}) \\&\qquad +\, \tau _2\sum _{i=1}^{t-1}({\mathbf {L}}\varvec{\beta }^{(m+i)} -\varvec{\eta }_2^{(m+i)})^{\mathrm {T}}({\mathbf {L}}\varvec{\beta }^{(m+t)} -\varvec{\eta }_2^{(m+t)}) + p_{\gamma }(|\varvec{\eta }_2^{(m+t)}|,\lambda _2 ) \\&\quad \le h^{(m+t)}. \end{aligned}$$

By the differentiable of ($S_n( \varvec{\beta },\varvec{\eta }_1,\varvec{\eta }_2, {\mathbf {v}}_1, {\mathbf {v}}_2 )$) with respect to $\varvec{\beta }$ and ($S_n( \varvec{\beta },\varvec{\eta }_1,\varvec{\eta }_2, {\mathbf {v}}_1, {\mathbf {v}}_2 )$) is convex with respect to $\mathbf {\varvec{\eta }}_1, \varvec{\eta }_2$, based on Theorem 4.1 in [12], the limits of $(\varvec{\beta }^{(m)}, \varvec{\eta }_1^{(m)}, \varvec{\eta }_2^{(m)} )$ exist, which denote as $(\varvec{\beta }^{*}, \varvec{\eta }_1^{*}, \varvec{\eta }_2^{*} )$. Therefore,

$$\begin{aligned} h^{*}= & {} \lim _{m\rightarrow \infty }h^{(m+1)}=\lim _{m\rightarrow \infty }h^{(m+t)}\\= & {} \inf _{\Xi (\varvec{\beta }^{*})} \biggl \{\frac{1}{2n}\left\| {\varvec{y}}-{\mathbf {X}}\varvec{\beta }^{*}\right\| ^{2}+{\mathbb {I}}_{{\mathcal {C}}}(\varvec{\eta }_1 ) + p_{\gamma }(|\varvec{\beta }^{*}|, \lambda _1 ) + p_{\gamma }(|\varvec{\eta }_2|,\lambda _2 )\biggr \}, \end{aligned}$$

and for any $t\ge 0$

$$\begin{aligned}&\lim _{m\rightarrow \infty }S_n\biggl (\varvec{\beta }^{(m+t)}, \varvec{\eta }^{(m+t)}_1,\varvec{\eta }_2^{(m+t)}, {\mathbf {v}}^{(m+t-1)}_1,{\mathbf {v}}_2^{(m+t-1)} \biggr ) \\&\quad =\frac{1}{2n}\left\| {\varvec{y}}-{\mathbf {X}}\varvec{\beta }^{*}\right\| ^{2}+ {\mathbb {I}}_{{\mathcal {C}}}(\varvec{\eta }_1^{*} ) + p_{\gamma }(|\varvec{\beta }^{*}|, \lambda _1 ) + p_{\gamma }(|\varvec{\eta }_2^{*}|,\lambda ) \\&\qquad +\, \lim _{m\rightarrow \infty }{\mathbf {v}}_1^{(m)\mathrm {T}}(\varvec{\beta }^{*}- \varvec{\eta }_1^{*})+(t-\frac{1}{2}) \tau _1 \Vert \varvec{\beta }^{*}- \varvec{\eta }_1^{*}\Vert ^{2} \\&\qquad +\,\lim _{m\rightarrow \infty }{\mathbf {v}}_2^{(m)\mathrm {T}}({\mathbf {L}}\varvec{\beta }^{*} - \varvec{\eta }_2^{*})+(t-\frac{1}{2}) \tau _2 \Vert {\mathbf {L}}\varvec{\beta }^{*}-\varvec{\eta }_2^{*}\Vert ^{2} \\&\quad \le h^{*}. \end{aligned}$$

Hence, $\lim _{m\rightarrow \infty }\Vert {\mathbf {r}}^{(m)}_1\Vert ^{2}=r_1^{*}= \Vert \varvec{\beta }^{*}- \varvec{\eta }_1^{*}\Vert ^2 = 0$ and $\lim _{m\rightarrow \infty }\Vert {\mathbf {r}}^{(m)}_2\Vert ^{2}=r_2^{*}= \Vert {\mathbf {L}}\varvec{\beta }^{*}- \varvec{\eta }_2^{*}\Vert ^2 = 0$.

By definition that $\varvec{\beta }^{(m+1)}$ is the minimizer of $S_n\biggl (\varvec{\beta }, \varvec{\eta }^{(m)}_1,\varvec{\eta }_2^{(m)},{\mathbf {v}}_1^{(m)}, {\mathbf {v}}_2^{(m)} \biggr )$, then

$$\begin{aligned} \partial S_n\biggl (\varvec{\beta }^{(m+1)}, \varvec{\eta }^{(m)}_1,\varvec{\eta }_2^{(m)} {\mathbf {v}}_1^{(m)}, {\mathbf {v}}_2^{(m)}\biggr )/\partial \varvec{\beta }= {\mathbf {0}}, \end{aligned}$$

and further on,

$$\begin{aligned}&\partial S_n(\varvec{\beta }^{(m+1)},\varvec{\eta }^{(m)}_1,\varvec{\eta }_2^{(m)}, {\mathbf {v}}_1^{(m)}, {\mathbf {v}}_2^{(m)})/\partial \varvec{\beta }\\&\quad =-\,\frac{1}{n}{\mathbf {X}}^{\mathrm {T}}({\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{(m+1)})+ \sum _{j=1}^{p}p'(|\beta _j^{(m+1)}|)\text{ sgn }(\beta _j^{(m+1)})+{\mathbf {v}}_1^{(m+1)}\\&\qquad +\,\tau _1(\varvec{\beta }^{(m+1)} - \varvec{\eta }_1^{(m)} ) +{\mathbf {L}}^{\mathrm {T}} {\mathbf {v}}_2^{(m)}+ {\mathbf {L}}^{\mathrm {T}}\tau _2( {\mathbf {L}}\varvec{\beta }^{(m+1)} - \varvec{\eta }_2^{(m)} ) \\&\quad =-\,\frac{1}{n}{\mathbf {X}}^{\mathrm {T}}({\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{(m+1)})+ \sum _{j=1}^{p}p'(|\beta _j^{(m+1)}|)\text{ sgn }(\beta _j^{(m+1)})\\&\qquad +\, {\mathbf {v}}_1^{(m+1)} + \tau _1(\varvec{\eta }_1^{(m+1)} - \varvec{\eta }_1^{(m)}) \\&\qquad +\, {\mathbf {L}}^{\mathrm {T}}({\mathbf {v}}_2^{(m+1)} +\tau _2(\varvec{\eta }_2^{(m+1)} - \varvec{\eta }_2^{(m)}) ), \end{aligned}$$

where the last equality is based on ${\mathbf {v}}_1^{(m+1)} = {\mathbf {v}}_1^{(m)} + \tau _1(\varvec{\beta }^{(m+1)} - \varvec{\eta }_1^{(m+1)} ), {\mathbf {v}}_2^{(m+1)} = {\mathbf {v}}_2^{(m)} + \tau _2({\mathbf {L}}\varvec{\beta }^{(m+1)} - \varvec{\eta }_2^{(m+1)} )$. Therefore,

$$\begin{aligned}&{\mathbf {s}}_1^{(m+1)}+{\mathbf {s}}_2^{(m+1)} \\&\quad =\tau _1(\varvec{\eta }_1^{(m+1)} - \varvec{\eta }_1^{(m)})+ {\mathbf {L}}^{\mathrm {T}}\tau _2(\varvec{\eta }_2^{(m+1)} - \varvec{\eta }_2^{(m)}) \\&\quad =\frac{1}{n}{\mathbf {X}}^{\mathrm {T}}({\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{(m+1)})- \sum _{j=1}^{p}p'(|\beta _j^{(m+1)}|)\text{ sgn }(\beta _j^{(m+1)})- {\mathbf {v}}_1^{(m+1)}-{\mathbf {L}}^{\mathrm {T}}{\mathbf {v}}_2^{(m+1)}. \end{aligned}$$

Since $\Vert \varvec{\beta }^{*} - \varvec{\eta }_1^{*} \Vert ^{2}= \Vert {\mathbf {L}}\varvec{\beta }^{*} - \varvec{\eta }_2^{*}\Vert ^2=0$,

$$\begin{aligned}&\lim _{m\rightarrow \infty }\partial S_n(\varvec{\beta }^{(m+1)}, \varvec{\eta }^{(m)}_1,\varvec{\eta }_2^{(m)}, {\mathbf {v}}_1^{(m)}, {\mathbf {v}}_2^{(m)})/\partial \varvec{\beta }\\&\quad =\lim _{m\rightarrow \infty }-\frac{1}{n}{\mathbf {X}}^{\mathrm {T}}({\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{(m+1)})+ \sum _{j=1}^{p}p'(|\beta _j^{(m+1)}|)\text{ sgn }(\beta _j^{(m+1)})\\&\qquad + {\mathbf {v}}_1^{(m+1)} + {\mathbf {L}}^{\mathrm {T}}{\mathbf {v}}_2^{(m+1)} \\&\quad =-\,\frac{1}{n}{\mathbf {X}}^{\mathrm {T}}({\varvec{y}}- {\mathbf {X}}\varvec{\beta }^{*})+ \sum _{j=1}^{p}p'(|\beta _j^{*}|)\text{ sgn }(\beta _j^{*})+ {\mathbf {v}}_1^{*} + {\mathbf {L}}^{\mathrm {T}}{\mathbf {v}}_2^{*}={\mathbf {0}}. \end{aligned}$$

Consequently, we have $\lim _{m\rightarrow \infty }{\mathbf {s}}_1^{(m+1)}+{\mathbf {s}}_2^{(m+1)}={\mathbf {0}}$.

1.2 B. Proof of Theorem 2.2

This proof is an adaptation to a prior constraint case of the proof given by [11]. Denote ${\mathbf {u}}= (u_1,\ldots , u_p )^{\mathrm {T}}$, and

$$\begin{aligned} V_n({\mathbf {u}}) =&\sum _{i=1}^{n}\{ (\epsilon _i-{\mathbf {u}}^{\mathrm {T}}{\mathbf {X}}_i/\sqrt{n})^2 - \epsilon _i^2 \} + \lambda _{n1} \sum _{j=1}^{p}\{ |\beta _{j}+ u_j/\sqrt{n} | - |\beta _{j}| \} \\&+ \lambda _{n2} \sum _{j=1}^{p-1}\{ |\beta _{j+1} - \beta _{j} + (u_{j+1} - u_{j})/\sqrt{n} | - |\beta _{j+1} - \beta _{j} | \} \\&+ \sum _{k=1}^{r}v_k (f_k(\varvec{\beta }+ {\mathbf {u}}/\sqrt{n}) - f_k(\varvec{\beta })) \end{aligned}$$

and note that $V_n$ is minimized at $\sqrt{n}(\widehat{\varvec{\beta }}_n -\varvec{\beta })$. First note that, finite dimensional p, based on the CLT

$$\begin{aligned} \sum _{i=1}^{n}\{ (\epsilon _i-{\mathbf {u}}^{\mathrm {T}}{\mathbf {X}}_i/\sqrt{n})^2 - \epsilon _i^2 \} \rightarrow _{d} -2{\mathbf {u}}^{\mathrm {T}}W + {\mathbf {u}}^{\mathrm {T}}C{\mathbf {u}}. \end{aligned}$$

For $L_1$ penalty,

$$\begin{aligned} \sum _{j=1}^{p}\{ |\beta _{j} + u_j/\sqrt{n} | - |\beta _{j}| \} \rightarrow \lambda _{01} \sum _{j=1}^{p}\{ u_j\text{ sgn }(\beta _j)&I(\beta _j \ne 0 )+ |u_j|I(\beta _j=0) \}, \\ \sum _{j=1}^{p-1}\{ |\beta _{j+1} - \beta _{j} + (u_{j+1} - u_{j})/\sqrt{n} | - |\beta _{j+1} - \beta _{j} | \}&\\ \rightarrow \lambda _{02} \sum _{j=1}^{p-1}\{ (u_{j+1} - u_{j})\text{ sgn }(\beta _{j+1} - \beta _{j})I(\beta _j \ne&\beta _{j+1} ) + |u_j - u_{j+1}|I(\beta _j=\beta _{j+1} ) \}, \end{aligned}$$

and

$$\begin{aligned} {\mathbf {v}}^{\mathrm {T}} ({\mathbf {f}}(\varvec{\beta }+ {\mathbf {u}}/\sqrt{n} ) - {\mathbf {f}}(\varvec{\beta }) ) \rightarrow {\mathbf {v}}_0^{\mathrm {T}} \frac{\partial {\mathbf {f}}(\varvec{\beta })}{\partial \varvec{\beta }}{\mathbf {u}}. \end{aligned}$$

Then, for finite dimensional p we have $V_n({\mathbf {u}}) \rightarrow _{d} V({\mathbf {u}})$, where

$$\begin{aligned} V({\mathbf {u}})&= -\,2{\mathbf {u}}^{\mathrm {T}}{\mathbf {W}}+ {\mathbf {u}}^{\mathrm {T}}C{\mathbf {u}}+ \lambda _{01} \sum _{j=1}^{p}\{ u_j\text{ sgn }(\beta _j)I(\beta _j \ne 0 )+ |u_j|I(\beta _j=0) \} \\&\quad +\, \lambda _{02} \sum _{j=1}^{p-1}\{ (u_{j+1} - u_{j})\text{ sgn }(\beta _{j+1} - \beta _{j})I(\beta _j \ne \beta _{j+1} ) \\&\quad +\, |u_j - u_{j+1}|I(\beta _j=\beta _{j+1} ) \} + {\mathbf {v}}_0^{\mathrm {T}} \frac{\partial {\mathbf {f}}(\varvec{\beta })}{\partial \varvec{\beta }}{\mathbf {u}}. \end{aligned}$$

Since $V_n({\mathbf {u}})$ is convex and $V({\mathbf {u}})$ has a unique minimum, it follows [5] that

$$\begin{aligned} \arg \min _{{\mathbf {u}}} V_n({\mathbf {u}}) = \sqrt{n}(\widehat{\varvec{\beta }}_n - \varvec{\beta })\rightarrow _{d} \arg \min _{{\mathbf {u}}} V({\mathbf {u}}). \end{aligned}$$