An active set Newton-CG method for ℓ₁ optimization

Abstract

In this paper, we investigate the active set identification technique of ISTA and provide some good properties. An active set Newton-CG method is then proposed for ${\ell }_1$ optimization. Under appropriate conditions, we show that the proposed method is globally convergent with some nonmonotone line search. The numerical comparisons with several state-of-art methods demonstrate the efficiency of the proposed method.

Introduction

Sparse recovery has received a lot of attention in the last decades. This is due to the fact that many applications such as signal or image processing and data mining/classification can be formulated as sparse recovery problems. Sparse recovery refers to recovering a sparse vector from a set of linear measurements, which is one of the most fundamental issues in compressed sensing (CS) [14]. Mathematically, it can be expressed as

$$\begin{gathered} \min {\Vert x\Vert }_0 \\ \text{s.t.}Ax=b, \end{gathered}$$

where ${\Vert x\Vert }_0$ counts the number of nonzero entries of x, $A\in {\mathcal{R}}^{m\times n}$ (usually $m\ll n$) is called a sensing matrix, and $b\in {\mathcal{R}}^m$ is a measurement vector. Problem (1.1) is NP-hard and difficult to solve in practice. To find the solution of (1.1), alternative models have been used, which include the basis pursuit (BP) problem

$$\begin{gathered} \min {\Vert x\Vert }_1 \\ \text{s.t.}Ax=b, \end{gathered}$$

the closely related ${\ell }_1$-regularized least squares problem

$$\underset{x\in {\mathcal{R}}^n}{\min }\frac{1}{2}{\Vert Ax-b\Vert }^2+\mu {\Vert x\Vert }_1,$$

the lasso problem

$$\begin{gathered} \min {\Vert Ax-b\Vert }^2 \\ \text{s.t.}{\Vert x\Vert }_1\le t, \end{gathered}$$

and the BP denoising problems

$$\begin{gathered} \min {\Vert x\Vert }_1 \\ \text{s.t.}\Vert Ax-b\Vert \le \sigma , \end{gathered}$$

where ${\Vert x\Vert }_1={\sum }_{i=1}^n|x_i|$ and $\Vert .\Vert$ denotes the Euclidean norm of vectors. The theory for penalty functions shows that the solution (1.3) approaches the solution of (1.2) as μ go to zero. The lasso problem (1.4) and the BP denoising problem (1.5) are equivalent to (1.3) for appropriate choices of the parameters t and σ. Thus it is very necessary and important to study problem (1.3). In this paper, we consider the more general ${\ell }_1$-regularized optimization problem

$$\min \varphi (x):=f(x)+\mu {\Vert x\Vert }_1,$$

where f is continuously differentiable and $\mu >0$.

Recently, there have been many different approaches for solving (1.6). A large class of first order algorithms for solving problem (1.6) is based on the iterative shrinkage thresholding algorithms (ISTA)[13], [17] or variants of ISTA. To accelerate convergence, a two-step ISTA (TwiISA) algorithm was developed in [5] and the sequential subspace optimization techniques was added to ISTA [16]. Beck and Teboulle [4] constructed a faster iterative shrinkage-thresholding algorithm called FISTA that keeps its simplicity of ISTA but possesses a better global rate of convergence. Furthermore, to improve practical performance result of the above methods, Wright et al. [35] introduced the sparse reconstruction by separable approximation (SpaRSA) for solving (1.6). Hager et al. [23] analyzed the convergence rate of SpaRSA and proposed an improved version of SpaRSA based on a cyclic version of the BB iteration and an adaptive choice for the reference function value in the line search. Hale et al. [22] proposed a fixed point continuation algorithm (FPC_BB) that embeds ISTA [13], [17] in a continuation strategy. Wen et al. [33], [34] proposed the FPC active set (FPC_AS) algorithm that combines shrinkage, subspace optimization and continuation technique. Cheng and Dai [10] proposed gradient-based methods with active set strategy to solve (1.6). Liang et al. [25] proposed a general forward-backwark splitting method which identifies the active manifold in a finite number of iterations and has a local linear convergence.

Various types of algorithms are also designed to solve the equivalent constrained optimization reformulation of problem (1.6) or (1.3). For instance, interior point methods [28], projected gradient method [18] and alternating direction method of multipliers SALSA [1], [6]. Other algorithms for the ${\ell }_1$ minimization include coordinate-wise descent methods [32], Bergman iterative regularization based methods [37], reduced-space algorithm [11], second-order methods [8], [9], [27], [36], [29], quasi-Newton methods [24], [26], gradient methods [30] for minimizing the more general function $J(x)+H(x)$, where J is nonsmooth, H is smooth, and both are convex, a smoothed penalty algorithm (SPA) [2]. We refer to papers [7], [14], [19], [8], [31], [26], [11], [25] for more advances in this area.

As pointed out by the authors in [33], [34], the algorithm ISTA is very efficient in obtaining a support superset, but it is not efficient in recovering signal values. This motivated them to develop an efficient algorithm FPC_AS. The algorithm FPC_AS is divided into two stages that are performed repeatedly. Specifically, at the first stage “nonmonotone line search (NMLS)”, a first-order iterative “shrinkage” method to estimate the support of the solution. At the second stages, “subspace optimization”, a smaller smooth subproblem is solved to recover the magnitudes of x. Theoretically, the authors in [33], [34] showed that there exists an accumulation of $x^{⁎}$ of $\{x^k\}$ generated by the algorithm FPC_AS, which is a stationary point of problem (1.6). Our approach is partially motivated by our belief that a second-order method should be faster than the first-order iterative shrinkage method. To accelerate the algorithm FPC_AS, we shall propose an active set Newton-CG to solve problem (1.6). We first investigate the active set identification technique of ISTA and provide some good properties. Based on the active set identification technique of ISTA, we propose the algorithm to solve (1.6). Specifically, the active variables and free variables are defined by the identification technique at each iteration. At each iteration, the same direction as that of the FPC_AS method [33], [34] at the first stage is used to update the active variables, while a second-order method is utilized for solving a smooth subproblem in order to update the free variables. Hence the method is distinct from the FPC_AS method [33], [34]. In addition, the nonmonotone line search [20] of the proposed method is different from that of the FPC_AS method. Under appropriate conditions, we show that every accumulation of $x^{⁎}$ of $\{x^k\}$ generated by the proposed algorithm is a stationary point of problem (1.6). Numerical experiments with logistic regression problems and compressive sensing problems demonstrate that the proposed approach is competitive with several known methods.

The remainder of the paper is organized as follows. Some notations and properties related to (1.6) are given in Section 2. In Section 3, we propose the algorithm. In Section 4, we establish the global convergence of the algorithm. Some numerical results are reported in Section 5 and conclusions are made in the last section.