Folded concave penalized sparse linear regression: sparsity, statistical performance, and algorithmic theory for local solutions

Abstract

This paper concerns the folded concave penalized sparse linear regression (FCPSLR), a class of popular sparse recovery methods. Although FCPSLR yields desirable recovery performance when solved globally, computing a global solution is NP-complete. Despite some existing statistical performance analyses on local minimizers or on specific FCPSLR-based learning algorithms, it still remains open questions whether local solutions that are known to admit fully polynomial-time approximation schemes (FPTAS) may already be sufficient to ensure the statistical performance, and whether that statistical performance can be non-contingent on the specific designs of computing procedures. To address the questions, this paper presents the following threefold results: (1) Any local solution (stationary point) is a sparse estimator, under some conditions on the parameters of the folded concave penalties. (2) Perhaps more importantly, any local solution satisfying a significant subspace second-order necessary condition (S$^3$ONC), which is weaker than the second-order KKT condition, yields a bounded error in approximating the true parameter with high probability. In addition, if the minimal signal strength is sufficient, the S$^3$ONC solution likely recovers the oracle solution. This result also explicates that the goal of improving the statistical performance is consistent with the optimization criteria of minimizing the suboptimality gap in solving the non-convex programming formulation of FCPSLR. (3) We apply (2) to the special case of FCPSLR with minimax concave penalty and show that under the restricted eigenvalue condition, any S$^3$ONC solution with a better objective value than the Lasso solution entails the strong oracle property. In addition, such a solution generates a model error (ME) comparable to the optimal but exponential-time sparse estimator given a sufficient sample size, while the worst-case ME is comparable to the Lasso in general. Furthermore, to guarantee the S$^3$ONC admits FPTAS.

Appendix

1.1 Some useful Lemmas

Lemma 6

For any $\mathbf x^{true}\in \mathfrak {R}^p$, $\mathbf A\in \mathfrak {R}^{n\times p}$, $W\in \mathfrak {R}^n$, $\mathbf b=\mathbf A\mathbf x^{true}+W$, consider f as defined in (3) with arbitrarily either $P_{\lambda }=P_{\lambda ,SCAD}$ or $P_{\lambda }=P_{\lambda ,MCP}$. Let $\mathbf x^0\in \mathfrak {R}^p$ be a feasible solution to (3). If $f(\mathbf x^0)$ satisfies that $f(\mathbf x^{0})\le f(\mathbf x^{lasso})$, where $\mathbf x^{lasso}$ is defined in (4) with the same problem data $\mathbf x^{true}$, $\mathbf A$, and $\mathbf b$ as (3) and with an arbitrary penalty parameter $\lambda _{lasso}>0$, then $f(\mathbf x^{0})-f(\mathbf x^{true})\le (\lambda _{lasso}+\lambda )\left| \mathbf x^{lasso} - \mathbf x^{true}\right| .$

Proof

Denote that $f_{lasso}(\mathbf x)=(2n)^{-1}\Vert \mathbf A\mathbf x-\mathbf b\Vert ^2+\sum _{i=1}^p\lambda _{lasso}\vert x_i\vert$ for any $\mathbf x=(x_i)\in \mathfrak {R}^p$.

Firstly, notice that by definition of $\mathbf x^{lasso}$ in (4), $f_{lasso}(\mathbf x^{lasso})\le f_{lasso}(\mathbf x^{true}).$ We then know that $(2n)^{-1}\Vert \mathbf A\mathbf x^{lasso}-\mathbf b\Vert ^2-(2n)^{-1}\Vert \mathbf A\mathbf x^{true}-\mathbf b\Vert ^2\le \sum _{i=1}^p\lambda _{lasso}\vert x_i^{true}\vert -\sum _{i=1}^p\lambda _{lasso}\vert x_i^{lasso}\vert \le \sum _{i=1}^p\lambda _{lasso}\vert x_i^{true}- x_i^{lasso}\vert = \lambda _{lasso}\vert \mathbf x^{true}- \mathbf x^{lasso}\vert$

Secondly, due to the concavity and differentiability of $P_\lambda (\cdot )$ on $\mathfrak {R}_+$ and the fact that $0\le P'_\lambda (\vert x\vert )\le \lambda$ for all $x\in \mathfrak {R}$, $\sum _{i=1}^{p}P_\lambda (\vert x_i^{lasso}\vert )- \sum _{i=1}^{p}P_\lambda (\vert x_i^{true}\vert )\le \sum _{i=1}^p P_\lambda '(\vert x_i^{true}\vert )\cdot \left( \vert x_i^{lasso}\vert -\vert x_i^{true}\vert \right) \le \sum _{i=1}^p P_\lambda '(\vert x_i^{true}\vert )\cdot \vert x_i^{lasso} - x_i^{true}\vert \le \lambda \left| \mathbf x^{lasso} - \mathbf x^{true}\right|$.

Combining the above and the assumption that $f(\mathbf x^{0})\le f(\mathbf x^{lasso})$, we know that $f(\mathbf x^{0})-f(\mathbf x^{true})\le f(\mathbf x^{lasso})-f(\mathbf x^{true}) =(2n)^{-1}\Vert \mathbf A\mathbf x^{lasso}-\mathbf b\Vert ^2+\sum _{i=1}^{p}P_\lambda (\vert x_i^{lasso}\vert )-(2n)^{-1}\Vert \mathbf A\mathbf x^{true}-\mathbf b\Vert ^2- \sum _{i=1}^{p}P_\lambda (\vert x_i^{true}\vert ) \le (\lambda _{lasso}+\lambda )\left| \mathbf x^{lasso} - \mathbf x^{true}\right| ,$ as claimed. $\square$

Lemma 7

Assume that Condition B holds with initial solution $\mathbf x^0\in \mathfrak {R}^p$. For any $\mathbf x^{true}\in \mathfrak {R}^p$, $\mathbf A\in \mathfrak {R}^{n\times p}$, $W\in \mathfrak {R}^n$, $\mathbf b=\mathbf A\mathbf x^{true}+W$, and for any $\mathbf x^*=(x_i^*)\in \mathfrak {R}^p$ that satisfies (i) S$^3$ONC to (3) with arbitrarily either $P_\lambda =P_{\lambda ,SCAD}$ or $P_\lambda =P_{\lambda ,MCP}$; and (ii) the inequality that $f(\mathbf x^*)\le f(\mathbf x^0)$, the following inequality holds: $(2n)^{-1}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2\le \,n^{-1}W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true}) +\min \left\{ \sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{true}\vert ,\,\sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{*}- x_i^{true}\vert \right\}$. If, in addition, $f(\mathbf x^*)\le f(\mathbf x^{true})+{\varGamma }$ for an arbitrary ${\varGamma }\ge 0$, then $\frac{1}{2n}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2\le \,\frac{1}{n}W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})+\min \left\{ \sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{true}\vert ,\,\sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{*}{-} x_i^{true}\vert ,\, P_{\lambda }(a\lambda )\cdot {(\vert \mathcal S\vert {-}\Vert \mathbf x^{*}\Vert _0)}{+}{\varGamma }\right\} .$

Proof

Notice that $\mathbf b=\mathbf A\mathbf x^{true}+W$. Then for any $\mathbf x=(x_i)\in \mathfrak {R}^p$: $(2n)^{-1}\Vert \mathbf A\mathbf x-\mathbf b\Vert ^2+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i\vert =(2n)^{-1}\Vert \mathbf A(\mathbf x-\mathbf x^{true})\Vert ^2+(2n)^{-1}W^\top W-n^{-1}W^\top \mathbf A(\mathbf x-\mathbf x^{true})+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i\vert$.

Since $\mathbf x^*$ satisfies S$^3$ONC, which implies FONC, we know that $\mathbf x^*\in \arg \inf \{ \frac{1}{2n}\Vert \mathbf A\mathbf x-\mathbf b\Vert ^2+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i\vert :\,\mathbf x\in \mathfrak {R}^p\}.$ Therefore, $\frac{1}{2n}\Vert \mathbf A\mathbf x^*-\mathbf b\Vert ^2+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i^*\vert \le \frac{1}{2n}\Vert \mathbf A\mathbf x^{true}-\mathbf b\Vert ^2+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i^{true}\vert$. Combining the above, we know that $(2n)^{-1}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2-n^{-1}W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})+\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i^{*}\vert \le \sum _{i=1}^pP'_\lambda (\vert x_i^{*}\vert )\vert x_i^{true}\vert$. Further invoking the definitions of $\mathbf x^{true}$ and $\mathcal S$ as well as triangular inequality and the fact that $P'_{\lambda }(\vert x\vert )\ge 0$ for any $x\in \mathfrak {R}$, we have $(2n)^{-1}{\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2} \le \, n^{-1}{W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})}+\sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{true}\vert -\sum _{i=1}^pP_{\lambda }'(\vert x^{*}_i\vert )\vert x_i^{*}\vert \le \, n^{-1}W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})+\sum _{i\in \mathcal S}P'_\lambda (\vert x_i^{*}\vert )\vert x_i^{true}- x_i^{*}\vert -\sum _{i\in \mathcal S^c}P_{\lambda }'(\vert x^{*}_i\vert )\vert x_i^{*}\vert$. We then obtain the claimed result in the first part of the lemma.

To show the second part, by assumption, $f(\mathbf x^*)\le f(\mathbf x^{true})+{\varGamma }$, we know $(2n)^{-1}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2-n^{-1}{W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})}+(2n)^{-1}{\Vert W\Vert ^2}+\sum _{i=1}^{p}P_{\lambda }(\vert x_i^*\vert )\le (2n)^{-1}{\Vert W\Vert ^2}+\sum _{i =1}^{p}P_{\lambda }(\vert x_i^{true}\vert )+{\varGamma }$. Noticing the fact that (i) $0\le P_{\lambda }(\vert x\vert )\le P_{\lambda }(a\lambda )$ for any $x\in \mathfrak {R}$, (ii) $P_{\lambda }(\vert 0\vert )=0$, and (iii) by definition of $\mathcal S^c$, $x_i^{true}=0$ for all $i\in \mathcal S^c$, we hence know $(2n)^{-1}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2-n^{-1}{W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})}\le P_{\lambda }(a\lambda )\cdot \vert \mathcal S\vert -\sum _{i=1}^{p}P_{\lambda }(\vert x_i^*\vert )+{\varGamma }$. Invoking Corollaries 3 and 4 under Condition B and the assumption that $f(\mathbf x^*)\le f(\mathbf x^0)$, we know that $x_i^*\ne 0\Longrightarrow \vert x_i^*\vert \ge a\lambda$. Also notice that $P_{\lambda }(\vert x\vert )=P_{\lambda }(a\lambda )$ for all $x\in \mathfrak {R}:\,\vert x\vert \ge a\lambda$. Therefore, the above implies $\sum _{i=1}^{p}P_{\lambda }(\vert x_i^*\vert )= {P_{\lambda }(a\lambda )\cdot \Vert \mathbf x^{*}\Vert _0}$ and $(2n)^{-1}\Vert \mathbf A(\mathbf x^{*}-\mathbf x^{true})\Vert ^2-n^{-1}{W^\top \mathbf A(\mathbf x^{*}-\mathbf x^{true})}\le P_{\lambda }(a\lambda )\cdot (\vert \mathcal S\vert -\Vert \mathbf x^{*}\Vert _0)+{\varGamma }$. Combined with the results from the first part of this lemma, we have the claimed result in the second part. $\square$

Lemma 8

Consider a subgaussian $\tilde{n}$-dimensional random vector $\tilde{W}$ in $\mathfrak {R}^{\tilde{n}}$ that satisfies $Prob[\vert \langle \tilde{W},\, \upsilon \rangle \vert \ge t]\le 2\exp \left( -{t^2}(2\sigma ^2)^{-1}\right)$. for any $\upsilon \in \mathfrak {R}^{\tilde{n}}:\, \Vert \upsilon \Vert =1$, then for any $V\in \mathfrak {R}^{\tilde{n}\times \tilde{n}}$ and ${\varSigma }_{v}=V^\top V$, $Prob[\Vert V \tilde{W}\Vert ^2\le \sigma ^2(\mathbf{Tr}({\varSigma }_v)+2\sqrt{\mathbf{Tr}({\varSigma }_v^2)t}+2\Vert {\varSigma }_v\Vert t)]\ge 1-\exp (-t)$ for any $t>0$, where $\mathbf{Tr}(\cdot )$ denotes the trace of a matrix.

Proof

Evident from Theorem 2.1 in [17]. $\square$