Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors

Abstract

Penalized regression is an attractive framework for variable selection problems. Often, variables possess a grouping structure, and the relevant selection problem is that of selecting groups, not individual variables. The group lasso has been proposed as a way of extending the ideas of the lasso to the problem of group selection. Nonconvex penalties such as SCAD and MCP have been proposed and shown to have several advantages over the lasso; these penalties may also be extended to the group selection problem, giving rise to group SCAD and group MCP methods. Here, we describe algorithms for fitting these models stably and efficiently. In addition, we present simulation results and real data examples comparing and contrasting the statistical properties of these methods.

Appendix

Before proving Proposition 1, we establish the groupwise convexity of all the objective functions under consideration. Note that for group SCAD and group MCP, although they contain nonconvex components and are not necessarily convex overall, the objective functions are still convex with respect to the variables in a single group.

Lemma 1

The objective function Q(β _j ) for regularized linear regression is a strictly convex function with respect to β _j for the group lasso, for group SCAD with γ>2, and for group MCP with γ>1.

Proof

Although Q(β _j ) is not differentiable, it is directionally twice differentiable everywhere. Let $\nabla_{\mathbf{d}}^{2} Q(\boldsymbol{\beta} _{j})$ denote the second derivative of Q(β _j ) in the direction d. Then the strict convexity of Q(β _j ) follows if $\nabla_{\mathbf{d}}^{2} Q(\boldsymbol{\beta} _{j})$ is positive definite at all β _j and for all d. Let ξ _∗ denote the infimum over β _j and d of the minimum eigenvalue of $\nabla_{\mathbf{d}}^{2} Q(\boldsymbol{\beta}_{j})$. Then, after some algebra, we obtain

$$\begin{aligned} \begin{array}{l@{\quad}l} \displaystyle\xi_* = 1 & \mathrm{Group}\ \mathrm{lasso} \\ \displaystyle\xi_* = 1 - \frac{1}{\gamma-1} & \mathrm{Group}\ \mathrm{SCAD} \\ \displaystyle\xi_* = 1 - \frac{1}{\gamma}& \mathrm{Group}\ \mathrm{MCP}. \end{array} \end{aligned}$$

These quantities are positive under the conditions specified in the lemma. □

We now proceed to the proof of Proposition 1.

Proof of Proposition 1

The descent property is a direct consequence of the fact that each updating step consists of minimizing Q(β) with respect to β _j . Lemma 1, along with the fact that the least squares loss function is continuously differentiable and coercive, provide sufficient conditions to apply Theorem 4.1 of Tseng (2001), thereby establishing that every limit point of {β ^(m)} is a stationary point of Q(β).

We further note that {β ^(m)} is guaranteed to converge to a unique limit point. Suppose that the sequence possessed two limit points, β′ and β″, such that for at least one group, $\boldsymbol{\beta}'_{j} \neq\boldsymbol{\beta}''_{j}$. For the transition β′→β″ to occur, the algorithm must pass through the point $(\boldsymbol{\beta}''_{j}, \boldsymbol{\beta}'_{-j})$. However, by Lemma 1, $\boldsymbol{\beta}'_{j}$ is the unique value minimizing Q(β _j |β _−j ). Thus, β′→β″ is not allowed by the group descent algorithm and {β ^(m)} possesses a single limit point. □

For Proposition 2, involving logistic regression, we proceed similarly, letting $R(\boldsymbol{\beta}|\tilde{\boldsymbol {\beta}})$ denote the majorizing approximation to Q(β) at $\tilde{\boldsymbol {\beta}}$.

Lemma 2

The majorizing approximation $R(\boldsymbol{\beta}_{j}|\tilde {\boldsymbol{\beta}})$ for regularized logistic regression is a strictly convex function with respect to β _j at all $\tilde{\boldsymbol{\beta}}$ for the group lasso, for group SCAD with γ>5, and for group MCP with γ>4.

Proof

Proceeding as in the previous lemma, and letting ξ _∗ denote the infimum over $\tilde{\boldsymbol{\beta}}$, β _j and d of the minimum eigenvalue of $\nabla_{\mathbf{d}}^{2} Q(\boldsymbol{\beta}_{j})$, we obtain

$$\begin{aligned} \begin{array}{l@{\quad}l} \displaystyle\xi_* = \frac{1}{4}& \mathrm{Group}\ \mathrm{lasso} \\ \displaystyle\xi_* = \frac{1}{4} - \frac{1}{\gamma-1}& \mathrm{Group}\ \mathrm{SCAD} \\ \displaystyle\xi_* = \frac{1}{4} - \frac{1}{\gamma}& \mathrm{Group}\ \mathrm{MCP}. \end{array} \end{aligned}$$

These quantities are positive under the conditions specified in the lemma. □

Proof of Proposition 2

The proposition makes two claims: descent with every iteration and convergence to a stationary point. To establish descent for logistic regression, we note that because L is twice differentiable, for any point η there exists a vector η ^∗∗ on the line segment joining η and η ^∗ such that

$$\begin{aligned} L(\boldsymbol{\eta}) &= L\bigl(\boldsymbol{\eta}^*\bigr) + \bigl(\boldsymbol{ \eta}-\boldsymbol{\eta}^*\bigr)^T \nabla L\bigl(\boldsymbol {\eta}^* \bigr) \\ &\quad{}+ \frac{1}{2} \bigl(\boldsymbol{\eta} -\boldsymbol{\eta }^* \bigr)^T \nabla^2 L\bigl(\boldsymbol{\eta}^{**} \bigr) \bigl(\boldsymbol{\eta }-\boldsymbol{\eta}^*\bigr) \\ &\leq\tilde{L}\bigl(\boldsymbol{\eta }|\boldsymbol{\eta}^*\bigr) \end{aligned}$$

where the inequality follows from the fact that v I−∇² L(η ^∗∗) is a positive semidefinite matrix. Descent now follows from the descent property of MM algorithms (Lange et al. 2000) coupled with the fact that each updating step consists of minimizing $R(\boldsymbol {\beta}_{j}|\tilde{\boldsymbol{\beta}})$.

To establish convergence to a stationary point, we note that if no elements of β tend to ±∞, then the descent property of the algorithm ensures that the sequence β ^(k) stays within a compact set and therefore possesses a limit point $\tilde{\boldsymbol{\beta }}$. Then, as in the proof of Proposition 1, Lemma 2 allows us to apply the results of Tseng (2001) and conclude that $\tilde {\boldsymbol{\beta}}$ must be a stationary point of $R(\boldsymbol{\beta}|\tilde {\boldsymbol{\beta}})$. Furthermore, because $R(\boldsymbol{\beta} |\tilde{\boldsymbol{\beta}})$ is tangent to Q(β) at $\tilde{\boldsymbol{\beta}}$, $\tilde{\boldsymbol{\beta}}$ must also be a stationary point of Q(β). □