GSDAR: a fast Newton algorithm for $\ell _0$ regularized generalized linear models with statistical guarantee

Abstract

We propose a fast Newton algorithm for $\ell _0$ regularized high-dimensional generalized linear models based on support detection and root finding. We refer to the proposed method as GSDAR. GSDAR is developed based on the KKT conditions for $\ell _0$-penalized maximum likelihood estimators and generates a sequence of solutions of the KKT system iteratively. We show that GSDAR can be equivalently formulated as a generalized Newton algorithm. Under a restricted invertibility condition on the likelihood function and a sparsity condition on the regression coefficient, we establish an explicit upper bound on the estimation errors of the solution sequence generated by GSDAR in supremum norm and show that it achieves the optimal order in finite iterations with high probability. Moreover, we show that the oracle estimator can be recovered with high probability if the target signal is above the detectable level. These results directly concern the solution sequence generated from the GSDAR algorithm, instead of a theoretically defined global solution. We conduct simulations and real data analysis to illustrate the effectiveness of the proposed method.

Appendix

In the appendix, we prove Lemma 1, Proposition 1 and Theorems 1 and 2.

1.1 Proof of Lemma 1

Proof

Let $L_\lambda (\varvec{\beta })={\mathcal {L}}(\varvec{\beta })+\lambda \Vert \varvec{\beta }\Vert _0$. Assume $\widehat{\varvec{\beta }}$ is a global minimizer of $L_\lambda (\varvec{\beta })$. Then by Theorem 10.1 in Rockafellar and Wets (2009), we have

$$\begin{aligned} \mathbf{0} \in \nabla {\mathcal {L}} (\widehat{\varvec{\beta }}) + \lambda \partial \Vert \widehat{\varvec{\beta }}\Vert _{0}, \end{aligned}$$

(7)

where $\partial \Vert \widehat{\varvec{\beta }}\Vert _{0}$ denotes the limiting subdifferential (see Definition 8.3 in Rockafellar and Wets (2009)) of $\Vert \cdot \Vert _{0}$ at $\widehat{\varvec{\beta }}$. Let $\widehat{\mathbf{d}}= -\nabla {\mathcal {L}}(\widehat{\varvec{\beta }})$ and define $G(\varvec{\beta }) = \frac{1}{2}\Vert \varvec{\beta }- (\widehat{\varvec{\beta }}+\widehat{\mathbf{d}})\Vert ^2 + \lambda \Vert \varvec{\beta }\Vert _0$. We recall that, from the definition of the limiting subdifferential of Definition 8.3 in Rockafellar and Wets (2009)), $\partial \Vert \widehat{\varvec{\beta }}\Vert _{0}$ satisfies that $\Vert \varvec{\beta }\Vert _0\ge \Vert \widehat{\varvec{\beta }}\Vert _0+\langle \partial \Vert \widehat{\varvec{\beta }}\Vert _{0}, \varvec{\beta }-\widehat{\varvec{\beta }}\rangle +o(\Vert \varvec{\beta }-\widehat{\varvec{\beta }}\Vert )$ for any $\varvec{\beta }\in {\mathbb {R}}^p$. (7) is equivalent to

$$\begin{aligned} \mathbf{0} \in \widehat{\varvec{\beta }}-(\widehat{\varvec{\beta }}+\widehat{\mathbf{d}}) + \lambda \partial \Vert \widehat{\varvec{\beta }}\Vert _{0}. \end{aligned}$$

Moreover, ${\widetilde{\varvec{\beta }}}$ being the minimizer of $G(\varvec{\beta })$ is equivalent to $\mathbf{0} \in \partial G({\widetilde{\varvec{\beta }}})$. Obviously, $\widehat{\varvec{\beta }}$ satisfies $\mathbf{0} \in \partial G(\widehat{\varvec{\beta }})$. Thus we deduce that $\widehat{\varvec{\beta }}$ is a KKT point of $G(\varvec{\beta })$. Then $\widehat{\varvec{\beta }}=H_{\lambda }(\widehat{\varvec{\beta }}+\widehat{\mathbf{d}})$ follows from the result that the KKT points of G coincide with its coordinate-wise minimizer (Huang et al. 2021). Conversely, suppose $\widehat{\varvec{\beta }}$ and $\widehat{\mathbf{d}}$ satisfy (2), then $\widehat{\varvec{\beta }}$ is a local minimizer of $L_{\lambda }(\varvec{\beta })$. To show $\widehat{\varvec{\beta }}$ is a local minimizer of $L_{\lambda }(\varvec{\beta })$, we can assume $\text{ h }$ is small enough and $\Vert \text{ h }\Vert _{\infty }<\sqrt{2\lambda }$. Then we will show $L_{\lambda }(\widehat{\varvec{\beta }}+\text{ h})\ge L_{\lambda }(\widehat{\varvec{\beta }})$ in two cases respectively.

First, we denote

$$\begin{aligned} {\widehat{A}}=\{i:|{\widehat{\beta }}_{i}+{\widehat{d}}_{i}|\ge \sqrt{2\lambda }\},\quad {\widehat{I}}=\{i:|{\widehat{\beta }}_{i}+{\widehat{d}}_{i}|<\sqrt{2\lambda }\}. \end{aligned}$$

By the definition of $H_{\lambda }(\cdot )$ and (2), we can conclude that $|{\widehat{\beta }}_{i}|\ge \sqrt{2\lambda }$ when $i \in {\widehat{A}}$ and $\widehat{\varvec{\beta }}_{{\widehat{I}}}=0$. Thus it yields that $\text{ supp }(\widehat{\varvec{\beta }})={\widehat{A}}$. Moreover, we also have $\widehat{\mathbf{d}}_{{\widehat{A}}}=[-\nabla {\mathcal {L}}(\widehat{\varvec{\beta }})]_{{\widehat{A}}}=0$, which is equivalent to $\widehat{\varvec{\beta }}_{{\widehat{A}}}\in \underset{\varvec{\beta }_{{\widehat{A}}}}{\text{ argmin }}~\widetilde{{\mathcal {L}}}(\varvec{\beta }_{{\widehat{A}}})$.

Case 1: $\text{ h}_{{\widehat{I}}}\ne 0$.

$$\begin{aligned}&\displaystyle \Vert \widehat{\varvec{\beta }}+\text{ h }\Vert _{0}=\Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}\Vert _{0}+\Vert \text{ h}_{{\widehat{I}}}\Vert _{0}, \\&\displaystyle \lambda \Vert \widehat{\varvec{\beta }}+\text{ h }\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}\Vert _{0}=\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}\Vert _{0} +\lambda \Vert \text{ h}_{{\widehat{I}}}\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}\Vert _{0}. \end{aligned}$$

Because $|{\widehat{\beta }}_{i}|\ge \sqrt{2\lambda }$ for $i\in {{\widehat{A}}}$ and $\Vert \text{ h }\Vert _{\infty }<\sqrt{2\lambda }$, we have

$$\begin{aligned}&\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}\Vert _{0}=0, \\&\lambda \Vert \widehat{\varvec{\beta }}+\text{ h }\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}\Vert _{0}=\lambda \Vert \text{ h}_{{\widehat{I}}}\Vert _{0}>\lambda . \end{aligned}$$

Therefore, we get

$$\begin{aligned}&L_{\lambda }(\widehat{\varvec{\beta }}+\text{ h})-L_{\lambda }(\widehat{\varvec{\beta }})\\&\quad =\sum _{i=1}^{n}[c(\text{ x}_{i}^{T}(\widehat{\varvec{\beta }}+\text{ h}))-c(\text{ x}_{i}^{T}\widehat{\varvec{\beta }})]-\text{ y}^T\mathbf{X}\text{ h }+\lambda \Vert \text{ h}_{{\widehat{I}}}\Vert _{0}\\&\quad>\sum _{i=1}^{n}[c(\text{ x}_{i}^{T}(\widehat{\varvec{\beta }}+\text{ h}))-c(\text{ x}_{i}^{T}\widehat{\varvec{\beta }})]-\text{ y}^T\mathbf{X}\text{ h }+\lambda \\&\quad >0. \end{aligned}$$

Let $m(\text{ h})=\sum _{i=1}^{n}[c(\text{ x}_{i}^{T}(\widehat{\varvec{\beta }}+\text{ h}))-c(\text{ x}_{i}^{T}\widehat{\varvec{\beta }})]-\text{ y}^{T}\mathbf{X}\text{ h }$, so $m(\text{ h})$ is a continuous function about $\text{ h }$. As $\text{ h }$ is small enough and $\Vert \text{ h }\Vert _{\infty }<\sqrt{2\lambda }$, then $m(\text{ h})+\lambda >0$. Thus the last inequality holds.

Case 2: $\text{ h}_{{\widehat{I}}}=0$.

$$\begin{aligned} \lambda \Vert \widehat{\varvec{\beta }}+\text{ h }\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}\Vert _{0}=\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}\Vert _{0}. \end{aligned}$$

As $|{\widehat{\beta }}_{i}|\ge \sqrt{2\lambda }$ for $i\in {{\widehat{A}}}$ and $\Vert \text{ h}_{{\widehat{A}}}\Vert _{\infty }<\sqrt{2\lambda }$, then we have

$$\begin{aligned} \lambda \Vert \widehat{\varvec{\beta }}+\text{ h }\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}\Vert _{0}=\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}\Vert _{0}-\lambda \Vert \widehat{\varvec{\beta }}_{{\widehat{A}}}\Vert _{0}=0, \end{aligned}$$

and

$$\begin{aligned}&L_{\lambda }(\widehat{\varvec{\beta }}+\text{ h})-L_{\lambda }(\widehat{\varvec{\beta }})\\&\quad =\sum _{i=1}^{n}[c(\text{ x}_{i}^{T}(\widehat{\varvec{\beta }}+\text{ h}))-c(\text{ x}_{i}^{T}\widehat{\varvec{\beta }})]-\text{ y}^T\mathbf{X}\text{ h }\\&\quad =\sum _{i=1}^{n}[c(\text{ x}_{i({\widehat{A}})}^{T}(\widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}))-c(\text{ x}_{i({\widehat{A}})}^{T}\widehat{\varvec{\beta }}_{{\widehat{A}}})]-\text{ y}^T\mathbf{X}_{{\widehat{A}}}\text{ h}_{{\widehat{A}}}\\&\quad =\sum _{i=1}^{n}[c(\text{ x}_{i({\widehat{A}})}^{T}(\widehat{\varvec{\beta }}_{{\widehat{A}}}+\text{ h}_{{\widehat{A}}}))]-\text{ y}^T\mathbf{X}_{{\widehat{A}}}(\widehat{\varvec{\beta }}_{{\widehat{A}}}+ \text{ h}_{{\widehat{A}}})\\&\qquad -\sum _{i=1}^{n}[c(\text{ x}_{i({\widehat{A}})}^{T}\widehat{\varvec{\beta }}_{{\widehat{A}}})]+\text{ y}^T\mathbf{X}_{{\widehat{A}}}\widehat{\varvec{\beta }}_{{\widehat{A}}}\\&\quad \ge 0. \end{aligned}$$

As known that $\widehat{\varvec{\beta }}_{{\widehat{A}}}\in \underset{\varvec{\beta }_{{\widehat{A}}}}{\text{ argmin }}~\widetilde{{\mathcal {L}}}(\varvec{\beta }_{{\widehat{A}}})$, so the last inequality holds. In summary, $\widehat{\varvec{\beta }}$ is a local minimizer of $L_{\lambda }(\widehat{\varvec{\beta }})$. $\square$

1.2 Proof of Proposition 1

Proof

Denote $D^{k}=-\left( {\mathcal {H}}^{k}\right) ^{-1} F\left( \text{ w}^{k}\right)$. Then

$$\begin{aligned} \text{ w}^{k+1}=\text{ w}^{k}-\left( {\mathcal {H}}^{k}\right) ^{-1} F\left( \text{ w}^{k}\right) \end{aligned}$$

can be recast as

$$\begin{aligned}&{\mathcal {H}}^{k} D^{k}=-F\left( \text{ w}^{k}\right) , \end{aligned}$$

(8)

$$\begin{aligned}&\text{ w}^{k+1}=\text{ w}^{k}+D^{k}. \end{aligned}$$

(9)

Partition $\text{ w}^{k}, D^{k}$ and $F\left( \text{ w}^{k}\right)$ according to $A^{k}$ and $I^{k}$ such that

$$\begin{aligned}&\text{ w}^{k}=\left( \begin{array}{c} \varvec{\beta }_{A^{k}}^{k} \\ \varvec{\beta }_{I^{k}}^{k} \\ \mathbf{d}_{A^{k}}^{k} \\ \mathbf{d}_{I^{k}}^{k} \end{array}\right) , \quad D^{k}=\left( \begin{array}{c} D_{A^{k}}^{\varvec{\beta }} \\ D_{I^{k}}^{\varvec{\beta }} \\ D_{A^{k}}^{\mathbf{d}} \\ D_{I^{k}}^{\mathbf{d}} \end{array}\right) , \end{aligned}$$

(10)

$$\begin{aligned}&F\left( \text{ w}^{k}\right) = \left[ \begin{array}{c} -\mathbf{d}_{A^{k}}^{k} \\ \varvec{\beta }_{I^{k}}^{k} \\ n[\nabla {\mathcal {L}}(\varvec{\beta }^k)]_{A^k}+n\mathbf{d}^k_{A^k} \\ n[\nabla {\mathcal {L}}(\varvec{\beta }^k)]_{I^k}+n\mathbf{d}^k_{I^k} \end{array} \right] . \end{aligned}$$

(11)

Substituting (10), (11) and ${\mathcal {H}}^{k}$ into (8), we have

$$\begin{aligned}&\displaystyle \left( \mathbf{d}_{A^{k}}^{k}+D_{A^{k}}^{\mathbf{d}}\right) ={\mathbf {0}}_{A^{k}}, \end{aligned}$$

(12)

$$\begin{aligned}&\displaystyle \varvec{\beta }_{I^{k}}^{k}+D_{I^{k}}^{\varvec{\beta }} ={\mathbf {0}}_{I^{k}}, \end{aligned}$$

(13)

$$\begin{aligned}&\displaystyle n\nabla ^2{\mathcal {L}}(\varvec{\beta }^k)\left( \begin{array}{c} D_{A^{k}}^{\varvec{\beta }} \\ D_{I^{k}}^{\varvec{\beta }} \end{array}\right) +n\left( \begin{array}{c} D_{A^{k}}^{\mathbf{d}} \\ D_{I^{k}}^{\mathbf{d}} \end{array}\right) =n[\nabla {\mathcal {L}}(\varvec{\beta }^k)]+n\mathbf{d}^k. \end{aligned}$$

(14)

It follows from (9) that

$$\begin{aligned} \left( \begin{array}{c} \varvec{\beta }_{A^{k}}^{k+1} \\ \varvec{\beta }_{I^{k}}^{k+1} \\ \mathbf{d}_{A^{k}}^{k+1} \\ \mathbf{d}_{I^{k}}^{k+1} \end{array}\right) =\left( \begin{array}{c} \varvec{\beta }_{A^{k}}^{k}+D_{A^{k}}^{\varvec{\beta }} \\ \varvec{\beta }_{I^{k}}^{k}+D_{I^{k}}^{\varvec{\beta }} \\ \mathbf{d}_{A^{k}}^{k}+D_{A^{k}}^{\mathbf{d}} \\ \mathbf{d}_{I^{k}}^{k}+D_{I^{k}}^{\mathbf{d}} \end{array}\right) . \end{aligned}$$

(15)

Substituting (15) into (12)–(14), we get (4) of Algorithm 1. This completes the proof. $\square$

1.3 Preparatory lemmas

The proofs of Theorems 1 and 2 are built on the following lemmas.

Lemma 2

Assume (C1) holds and $\Vert \varvec{\beta }^{*}\Vert _{0}=K\le T$. Denote $B^k = A^{k}\backslash A^{k-1}$. Then,

$$\begin{aligned} \Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^{k})\Vert _1\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^{k})\Vert _{\infty }\ge 2L\zeta [{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)], \end{aligned}$$

where $\zeta =\frac{|B^k|}{|B^k|+|A^*\backslash A^{k-1}|}$.

Proof

Obviously, this lemma holds if $A^{k}=A^{k-1}$ or ${\mathcal {L}}(\varvec{\beta }^k)\le {\mathcal {L}}(\varvec{\beta }^*)$. So we only prove the lemma by assuming $A^{k}\ne A^{k-1}$ and ${\mathcal {L}}(\varvec{\beta }^k)>{\mathcal {L}}(\varvec{\beta }^*)$. The condition (C1) indicates

$$\begin{aligned}&{\mathcal {L}}(\varvec{\beta }^{*})-{\mathcal {L}}(\varvec{\beta }^k)-\langle \nabla {\mathcal {L}}(\varvec{\beta }^k),{\varvec{\beta }^*-\varvec{\beta }^k}\rangle \\&\quad \ge \frac{L}{2}\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _1\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _{\infty }. \end{aligned}$$

Hence,

$$\begin{aligned}&\langle -\nabla {\mathcal {L}}(\varvec{\beta }^k),{\varvec{\beta }^*-\varvec{\beta }^k}\rangle \\&\quad =\langle \nabla {\mathcal {L}}(\varvec{\beta }^k),-\varvec{\beta }^*\rangle \\&\quad \ge \frac{L}{2}\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _1\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _{\infty }+{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^{*})\\&\quad \ge \sqrt{2L}\sqrt{\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _1\big \Vert \varvec{\beta }^*-\varvec{\beta }^k\big \Vert _{\infty }}\sqrt{{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^{*})}.\\ \end{aligned}$$

From the definition of $A^{k}$ and $A^*$, it is known that $B^k$ contains the first $|B^k|$-largest elements (in absolute value) of $\nabla {\mathcal {L}}(\varvec{\beta }^k)$, and $\text{ supp }(\nabla {\mathcal {L}}(\varvec{\beta }^k))\bigcap \text{ supp }(\varvec{\beta }^*)=A^{*}\backslash A^{k-1}$. Thus, we have

$$\begin{aligned} \langle \nabla {\mathcal {L}}(\varvec{\beta }^k),-\varvec{\beta }^*\rangle\le & {} \frac{1}{\sqrt{\zeta }}\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _2\Vert \varvec{\beta }_{A^*\backslash A^{k-1}}^{*}\Vert _2\\= & {} \frac{1}{\sqrt{\zeta }}\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _2\Vert (\varvec{\beta }^{*}-\varvec{\beta }^k)_{A^*\backslash A^{k-1}}\Vert _2\\\le & {} \frac{1}{\sqrt{\zeta }}\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _2\Vert \varvec{\beta }^{*}-\varvec{\beta }^k\Vert _2\\\le & {} \frac{1}{\sqrt{\zeta }}\sqrt{\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _1\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _{\infty }}\\&\times \sqrt{\Vert \varvec{\beta }^{*}-\varvec{\beta }^k\Vert _1\Vert \varvec{\beta }^{*}-\varvec{\beta }^k\Vert _{\infty }}. \end{aligned}$$

Therefore,

$$\begin{aligned}&{\sqrt{2L}\sqrt{{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^{*})}}\\&\quad \le \frac{1}{\sqrt{\zeta }} \sqrt{\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^k)\Vert _1\Vert \nabla _{B^{k}} {\mathcal {L}}(\varvec{\beta }^k)\Vert _{\infty }}. \end{aligned}$$

In summary,

$$\begin{aligned} \Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^{k})\Vert _1\Vert \nabla _{B^k}{\mathcal {L}}(\varvec{\beta }^{k})\Vert _{\infty }\ge 2L\zeta [{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)]. \end{aligned}$$

$\square$

Lemma 3

Assume (C1) holds with $0<U<\frac{1}{T}$, and $K\le T$ in Algorithm 1. Then before Algorithm 1 terminates,

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^*)\le \xi [{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)], \end{aligned}$$

where $\xi =1-\frac{2L(1-TU)}{T(1+K)}\in (0,1)$.

Proof

Let $\Delta ^{k}=\varvec{\beta }^k-\nabla {\mathcal {L}}(\varvec{\beta }^{k})$. The condition of (C1) indicates

$$\begin{aligned}&{\mathcal {L}}(\Delta ^{k+1}|_{A^{k+1}})-{\mathcal {L}}(\varvec{\beta }^{k+1})\le \langle \nabla {\mathcal {L}}(\varvec{\beta }^{k+1}),\Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\rangle \\&\quad \quad \quad \quad +\frac{U}{2}\big \Vert \Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\big \Vert _1\big \Vert \Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\big \Vert _{\infty }. \end{aligned}$$

On the one hand, by the definition of $\varvec{\beta }^{k+1}$ and $\nabla {\mathcal {L}}(\varvec{\beta }^{k+1})$, we have

$$\begin{aligned}&\langle \nabla {\mathcal {L}}(\varvec{\beta }^{k+1}), \Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\rangle \\&\quad =\langle \nabla {\mathcal {L}}(\varvec{\beta }^{k+1}), \Delta ^{k+1}|_{A^{k+1}}\rangle \\&\quad =\langle \nabla _{A^{k+1}}{\mathcal {L}}(\varvec{\beta }^{k+1}), \Delta _{A^{k+1}}^{k+1}\rangle \\&\quad =\langle \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1}), \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\rangle . \end{aligned}$$

Further, we also have

$$\begin{aligned}&\big \Vert \Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\big \Vert _1\\&\quad =\big \Vert \Delta ^{k+1}|_{A^{k+1}\backslash A^{k}}+\Delta ^{k+1}|_{A^{k+1}\bigcap A^{k}}\\&\quad \quad -\varvec{\beta }^{k+1}|_{A^{k+1}\bigcap A^{k}}-\varvec{\beta }^{k+1}|_{A^{k}\backslash A^{k+1}}\big \Vert _1\\&\quad =\big \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\big \Vert _1+\big \Vert \Delta _{A^{k+1}\bigcap A^{k}}^{k+1}-\varvec{\beta }_{A^{k+1}\bigcap A^{k}}^{k+1}\big \Vert _1\\&\quad \quad +\big \Vert \varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}\big \Vert _1\\&\quad =\big \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\big \Vert _1+\big \Vert \varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}\big \Vert _1, \end{aligned}$$

and

$$\begin{aligned}&\big \Vert \Delta ^{k+1}|_{A^{k+1}}-\varvec{\beta }^{k+1}\big \Vert _{\infty }\\&\quad =\big \Vert \Delta ^{k+1}|_{A^{k+1}\backslash A^{k}}+\Delta ^{k+1}|_{A^{k+1}\bigcap A^{k}}\\&\quad \quad -\varvec{\beta }^{k+1}|_{A^{k+1}\bigcap A^{k}}-\varvec{\beta }^{k+1}|_{A^{k}\backslash A^{k+1}}\big \Vert _{\infty }\\&\quad =\big \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\big \Vert _{\infty }\bigvee \big \Vert \varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}\big \Vert _{\infty }, \end{aligned}$$

where $a \bigvee b=\max \{a,b\}$. By the definition of $A^k$, $A^{k+1}$ and $\varvec{\beta }^{k+1}$, we know that

$$\begin{aligned} |A^{k}\backslash A^{k+1}|=|A^{k+1}\backslash A^{k}|,\quad \Delta _{A^{k}\backslash A^{k+1}}^{k+1}=\varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}. \end{aligned}$$

By the definition of $A^{k+1}$, we can conclude that

$$\begin{aligned}&\Vert \Delta _{A^{k}\backslash A^{k+1}}^{k+1}\Vert _1=\Vert \varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}\Vert _1\le \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\Vert _1, \\&\Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\Vert _{\infty }\bigvee \Vert \varvec{\beta }_{A^{k}\backslash A^{k+1}}^{k+1}\Vert _{\infty }=\Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\Vert _{\infty }. \end{aligned}$$

Due to $-\nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1})=\Delta _{A^{k+1}\backslash A^{k}}^{k+1}$ and $U<\frac{1}{T}$, hence we can deduce that

$$\begin{aligned}&{{\mathcal {L}}(\Delta ^{k+1}|_{A^{k+1}})-{\mathcal {L}}(\varvec{\beta }^{k+1})}\\&\quad \le \langle \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1}),\Delta _{A^{k+1}\backslash A^{k}}^{k+1}\rangle +U\big \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\big \Vert _1\big \Vert \Delta _{A^{k+1}\backslash A^{k}}^{k+1}\big \Vert _{\infty }\\&\quad \le -(1/T-U)\big \Vert \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1})\big \Vert _1 \times \big \Vert \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1})\big \Vert _{\infty }. \end{aligned}$$

By the definition of $\varvec{\beta }^{k+1}$, we have

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^{k})\le & {} {\mathcal {L}}(\Delta ^{k}|_{A^{k}})-{\mathcal {L}}(\varvec{\beta }^{k})\\\le & {} -(1/T-U)\big \Vert \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1})\big \Vert _1 \times \big \Vert \nabla _{A^{k+1}\backslash A^{k}}{\mathcal {L}}(\varvec{\beta }^{k+1})\big \Vert _{\infty }. \end{aligned}$$

Moreover, $\frac{|A^*\backslash A^{k-1}|}{|B^k|}\le K$. By Lemma 2, we have

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^{k})\le -\frac{2L(1-TU)}{T(1+K)}[{\mathcal {L}}(\varvec{\beta }^{k})-{\mathcal {L}}(\varvec{\beta }^{*})]. \end{aligned}$$

Therefore, we have

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^*)\le \xi [{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)], \end{aligned}$$

where $\xi =1-\frac{2L(1-TU)}{T(1+K)}\in (0,1)$. $\square$

Lemma 4

Assume ${\mathcal {L}}$ satisfies (C1) and

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^{*})\le \xi [{\mathcal {L}}(\varvec{\beta }^{k})-{\mathcal {L}}(\varvec{\beta }^{*})] \end{aligned}$$

for all $k\ge 0$. Then,

$$\begin{aligned} \begin{aligned} \Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }&\le \sqrt{(K+T)(1+\frac{U}{L})}(\sqrt{\xi })^k\Vert \varvec{\beta }^0-\varvec{\beta }^*\Vert _{\infty }\\&\quad +\frac{2}{L}\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }. \end{aligned} \end{aligned}$$

(16)

Proof

If $\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }< \frac{2\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }}{L}$, then (16) holds, so we only consider the case that $\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }\ge \frac{2\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }}{L}$. On the one hand, ${\mathcal {L}}$ satisfies (C1), then

$$\begin{aligned}&{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)\\&\quad \ge \langle \nabla {\mathcal {L}}(\varvec{\beta }^*),\varvec{\beta }^k-\varvec{\beta }^*\rangle +\frac{L}{2}\big \Vert \varvec{\beta }^k-\varvec{\beta }^*\big \Vert _{1}\big \Vert \varvec{\beta }^k-\varvec{\beta }^*\big \Vert _{\infty }\\&\quad \ge -\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _1+\frac{L}{2}\big \Vert \varvec{\beta }^k-\varvec{\beta }^*\big \Vert _1\big \Vert \varvec{\beta }^k-\varvec{\beta }^*\big \Vert _{\infty }. \end{aligned}$$

Due to $\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }\ge \frac{2\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }}{L}$, then

$$\begin{aligned} (\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _1-\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty })\left( \frac{L}{2}\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }-\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }\right) \ge 0. \end{aligned}$$

Further, we can get

$$\begin{aligned} \frac{L}{2}\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }^2-\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }-[{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)]\le 0, \end{aligned}$$

which is univariate quadratic inequality about $\Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }$. Thus, by simple computation, we can get

$$\begin{aligned} \Vert \varvec{\beta }^k-\varvec{\beta }^*\Vert _{\infty }\le \sqrt{\frac{2\max \{{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*),0\}}{L}}+\frac{2\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }}{L}. \end{aligned}$$

(17)

On the other hand, because ${\mathcal {L}}$ satisfies (C1), then

$$\begin{aligned}&{\mathcal {L}}(\varvec{\beta }^0)-{\mathcal {L}}(\varvec{\beta }^*)\\&\quad \le \langle \nabla {\mathcal {L}}(\varvec{\beta }^*),\varvec{\beta }^0-\varvec{\beta }^*\rangle +\frac{U}{2}\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _1\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _{\infty }\\&\quad \le \big \Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\big \Vert _{\infty }\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _1+\frac{U}{2}\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _1\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _{\infty }\\&\quad \le (K+T)\big \Vert \varvec{\beta }^0-\varvec{\beta }^{*}\big \Vert _{\infty }(\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\big \Vert _{\infty }+\frac{U}{2}\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _{\infty }). \end{aligned}$$

Then, we can get

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k})-{\mathcal {L}}(\varvec{\beta }^{*})&\le \xi [{\mathcal {L}}(\varvec{\beta }^{k-1})-{\mathcal {L}}(\varvec{\beta }^{*})]\\&\le \xi ^{k}[{\mathcal {L}}(\varvec{\beta }^{0})-{\mathcal {L}}(\varvec{\beta }^{*})]\\&\le \xi ^k(K+T)\big \Vert \varvec{\beta }^0-\varvec{\beta }^{*}\big \Vert _{\infty }\\&\quad \times (\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\big \Vert _{\infty }+\frac{U}{2}\big \Vert \varvec{\beta }^0-\varvec{\beta }^*\big \Vert _{\infty })\\&\le \frac{\xi ^k (L+U)(K+T)}{2}\big \Vert \varvec{\beta }^0-\varvec{\beta }^{*}\big \Vert _{\infty }^2. \end{aligned}$$

Hence, by (17), we have

$$\begin{aligned} \Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }&\le \sqrt{(K+T)(1+\frac{U}{L})}(\sqrt{\xi })^k\Vert \varvec{\beta }^0-\varvec{\beta }^*\Vert _{\infty }\\&\quad +\frac{2}{L}\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }. \end{aligned}$$

$\square$

Lemma 5

(Proof of Corollary 2 in Loh and Wainwright (2015)). Assume $x_{ij}^{,}s$ are sub-Gaussian and $n > rsim \log (p)$, then there exists universal constants $(c_1,c_2,c_3)$ with $0<c_i<\infty$, $i=1,2,3$ such that

$$\begin{aligned} P\left( \Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }\ge c_1\sqrt{\frac{\log (p)}{n}}\right) \le c_2\exp (-c_3\log (p)). \end{aligned}$$

1.4 Proof of Theorem 1

Proof

By Lemma 3, we have

$$\begin{aligned} {\mathcal {L}}(\varvec{\beta }^{k+1})-{\mathcal {L}}(\varvec{\beta }^*)\le \xi [{\mathcal {L}}(\varvec{\beta }^k)-{\mathcal {L}}(\varvec{\beta }^*)], \end{aligned}$$

where

$$\begin{aligned} \xi =1-\frac{2L(1-TU)}{T(1+K)}\in (0,1). \end{aligned}$$

So the conditions of Lemma 4 are satisfied. Taking $\varvec{\beta }^0 = \mathbf{0}$, we can get

$$\begin{aligned}&\Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }\\&\quad \le \sqrt{(K+T)(1+\frac{U}{L})}(\sqrt{\xi })^k\Vert \varvec{\beta }^*\Vert _{\infty } +\frac{2}{L}\Vert \nabla {\mathcal {L}}(\varvec{\beta }^*)\Vert _{\infty }. \end{aligned}$$

By Lemma 5, then there exists universal constants $(c_1,c_2,c_3)$ defined in Lemma 5, with at least probability $1-c_2\exp (-c_3\log (p))$, we have

$$\begin{aligned}&\Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }\nonumber \\&\quad \le \sqrt{(K+T)(1+\frac{U}{L})}(\sqrt{\xi })^k\Vert \varvec{\beta }^*\Vert _{\infty } +\frac{2c_1}{L} \sqrt{\frac{\log (p)}{n}}. \end{aligned}$$

(18)

Some algebra shows that

$$\begin{aligned} \Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }\le {\mathcal {O}}(\sqrt{\frac{\log (p)}{n}}) \end{aligned}$$

by taking $k \ge {\mathcal {O}}(\log _{\frac{1}{\xi }} \frac{n}{\log (p)} )$ in (18). Then, the proof is complete. $\square$

1.5 Proof of Theorem 2

Proof

(18) and assumption (C2) and some algebra shows that that

$$\begin{aligned}&\Vert \varvec{\beta }^{k}-\varvec{\beta }^{*}\Vert _{\infty }\\&\le \sqrt{(K+T)(1+\frac{U}{L})}(\sqrt{\xi })^k\Vert \varvec{\beta }^*\Vert _{\infty } +\frac{2}{3}\Vert \varvec{\beta }^*_{A^*}\Vert _{\min }\\&< \Vert \varvec{\beta }^*_{A^*}\Vert _{\min }, \end{aligned}$$

if $k>\log _{\frac{1}{\xi }} 9 (T+K)(1+\frac{U}{L})r^2.$ This implies that $A^* \subseteq A^k$. $\square$