A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems

Abstract

In this paper, we describe and establish iteration-complexity of two accelerated composite gradient (ACG) variants to solve a smooth nonconvex composite optimization problem whose objective function is the sum of a nonconvex differentiable function f with a Lipschitz continuous gradient and a simple nonsmooth closed convex function h. When f is convex, the first ACG variant reduces to the well-known FISTA for a specific choice of the input, and hence the first one can be viewed as a natural extension of the latter one to the nonconvex setting. The first variant requires an input pair (M, m) such that f is m-weakly convex, $\nabla f$ is M-Lipschitz continuous, and $m \le M$ (possibly $m<M$), which is usually hard to obtain or poorly estimated. The second variant on the other hand can start from an arbitrary input pair (M, m) of positive scalars and its complexity is shown to be not worse, and better in some cases, than that of the first variant for a large range of the input pairs. Finally, numerical results are provided to illustrate the efficiency of the two ACG variants.

Supplementary results

This section provides a bound on the quantity $\min _{0 \le i \le k-1} \Vert y_{i+1}-\tilde{x}_i\Vert ^2$ for the case in which the parameter $m_0$ of the ADAP-NC-FISTA satisfies $m_0\ge {\bar{m}}$. Note that an alternative bound on this quantity has already been developed in Proposition 3.3 for any $m_0>0$.

Proposition A.1

For every $k\ge 1$, for $m_0\ge {\bar{m}}$, we have

$$\begin{aligned} \frac{1}{10}\left( \sum _{i=0}^{k-1}A_{i+1}\right) \min _{0 \le i \le k-1} \Vert y_{i+1} - {\tilde{x}}_i \Vert ^2\le & {} 2{\lambda }_0 A_0 \left( \phi \left( y_0\right) -\phi _*\right) +\Vert x_0-x^*\Vert ^2\\&+{\lambda }_0D_h^2 \left( 2m_0+2m_0 k+ {\bar{m}}\sum _{i=0}^{k-1}a_i\right) . \end{aligned}$$

Proof

Using the assumption of the lemma that $m_0\ge {\bar{m}}$, the facts that $a_i \ge 2$ for $i\ge 0$ from the fourth remark following SUB$(\theta ,{\lambda },m)$, and $\{{\lambda }_i\}$ is non-increasing from Lemma 3.1(d), we have

$$\begin{aligned} \left( {\bar{m}}+\frac{2m_0}{a_i}\right) {\lambda }_{i+1}\le m_0 \left( 1 +\frac{2}{a_i}\right) {\lambda }_{i+1} \le 2m_0{\lambda }_i. \end{aligned}$$

(63)

The above inequality implies that (34) is always satisfied with $m=m_0$ and ${\lambda }={\lambda }_{i+1}$. Hence, $m_k$ is never updated in SUB$(\theta ,{\lambda },m)$, i.e., $m_i=m_0$, for $i\ge 0$. Using similar arguments as in the proof of Lemma 2.3, we conclude that for every $i \ge 0$ and $u \in \Omega$,

$$\begin{aligned}&2{\lambda }_{i+1} A_{i+1} \phi \left( y_{i+1}\right) + \left( 2m_0{\lambda }_{i+1}+1\right) \Vert u - x_{i+1} \Vert ^2 + \left( 1-{\lambda }_{i+1} {{{\mathcal {C}}}}_{i+1}\right) A_{i+1} \Vert y_{i+1} - {\tilde{x}}_i \Vert ^2 \nonumber \\&\quad \le 2{\lambda }_{i+1} A_i \gamma _i\left( y_i\right) + 2{\lambda }_{i+1} a_i \gamma _i(u) + \Vert u - x_i \Vert ^2, \end{aligned}$$

(64)

where

$$\begin{aligned} \gamma _i(u) := \tilde{\gamma }_i\left( y_{i+1}\right) + \frac{1}{{\lambda }_{i+1}}\langle {\tilde{x}}_i - y_{i+1}, u - y_{i+1} \rangle + \frac{m_0}{a_i} \Vert u - y_{i+1} \Vert ^2 \end{aligned}$$

and

$$\begin{aligned} \tilde{\gamma }_i(u) := \ell _f\left( u;{\tilde{x}}_i\right) + h(u) + \frac{m_0}{a_i} \Vert u - {\tilde{x}}_i \Vert ^2. \end{aligned}$$

(65)

As in Lemma 2.2(a), we have $\gamma _i(u)\le \tilde{\gamma }_i(u)$ for every $u\in \mathrm {dom}\,h$. Hence, it follows from (65) and (4) that for every $k\ge 0$ and $u \in \mathrm {dom}\,h$, we have

$$\begin{aligned} \gamma _i(u)-\phi (u)&\le \tilde{\gamma }_i(u)-\phi (u) =\ell _f\left( u;\tilde{x}_i\right) -f(u)+\frac{m_0}{a_i}\Vert u-\tilde{x}_i\Vert ^2 \le \frac{1}{2}\left( {\bar{m}}+\frac{2m_0}{a_i}\right) \Vert u-\tilde{x}_i\Vert ^2. \end{aligned}$$

(66)

Taking $u=x^*$, and using (64), (23), (66), (63), Lemma 3.1(c), and the facts that $x_0=y_0$, ${\lambda }_i\le {\lambda }_0$ and $\phi (y_i)\ge \phi _*$ for $i\ge 0$, we conclude that for every $0 \le i \le k-1$,

$$\begin{aligned}&0.1A_{i+1} \Vert y_{i+1} - {\tilde{x}}_i \Vert ^2 - 2{\lambda }_{i} A_{i}\left( \phi \left( y_{i}\right) - \phi _*\right) - \Vert x^* - x_{i} \Vert ^2 \nonumber \\&\qquad + 2{\lambda }_{i+1}A_{i+1} \left( \phi \left( y_{i+1}\right) - \phi _*\right) + \left( 2m_0{\lambda }_{i+1}+1\right) \Vert x^* - x_{i+1} \Vert ^2 \nonumber \\&\quad \le 2{\lambda }_{i+1} A_i\left( \gamma _i\left( y_i\right) - \phi \left( y_i\right) \right) + 2{\lambda }_{i+1} a_i\left( \gamma _i\left( x^*\right) - \phi _* \right) +2\left( {\lambda }_{i+1}-{\lambda }_{i}\right) A_{i}\left( \phi (y_{i}) - \phi _*\right) \nonumber \\&\quad \le {\lambda }_{i+1}\left( {\bar{m}} + \frac{2m_0}{a_i} \right) \left( A_i\Vert y_i - \tilde{x}_i \Vert ^2 + a_i\Vert x^* - \tilde{x}_i \Vert ^2 \right) \nonumber \\&\quad \le {\lambda }_{i+1}\left( {\bar{m}} + \frac{2m_0}{a_i} \right) \left( \Vert x^* - x_i \Vert ^2 + a_i D_h^2 \right) \nonumber \\&\quad \le 2m_0{\lambda }_{i} \Vert x_i-x^* \Vert ^2 + \left( {\bar{m}} a_i+2m_0\right) {\lambda }_{i+1} D_h^2 \nonumber \\&\quad \le 2m_0{\lambda }_{i} \Vert x_i-x^* \Vert ^2 + \left( {\bar{m}} a_i+2m_0\right) {\lambda }_0 D_h^2. \end{aligned}$$

The conclusion is obtained by rearranging terms and summing the above inequality from $i=0$ to $k-1$. $\square$