Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

Abstract

We study linear convergence of some first-order methods such as the proximal gradient method (PGM), the proximal alternating linearized minimization (PALM) algorithm and the randomized block coordinate proximal gradient method (R-BCPGM) for minimizing the sum of a smooth convex function and a nonsmooth convex function. We introduce a new analytic framework based on the error bound/calmness/metric subregularity/bounded metric subregularity. This variational analysis perspective enables us to provide some concrete sufficient conditions for linear convergence and applicable approaches for calculating linear convergence rates of these first-order methods for a class of structured convex problems. In particular, for the LASSO, the fused LASSO and the group LASSO, these conditions are satisfied automatically, and the modulus for the calmness/metric subregularity is computable. Consequently, the linear convergence of the first-order methods for these important applications is automatically guaranteed and the convergence rates can be calculated. The new perspective enables us to improve some existing results and obtain novel results unknown in the literature. Particularly, we improve the result on the linear convergence of the PGM and PALM for the structured convex problem with a computable error bound estimation. Also for the R-BCPGM for the structured convex problem, we prove that the linear convergence is ensured when the nonsmooth part of the objective function is the group LASSO regularizer.

Appendix

Proof of Theorem 1

Since $L_{i_k}$ is the Lipschitz constant of $\nabla _{i_k} f$, $c^k_i \ge L_i$ and $x^{k+1}_j = x^k_j, \forall j\neq i_k$, we have

$$ f(x^{k+1}) \le f(x^{k}) + \left\langle \nabla_{i_{k}} f(x^{k}), x^{k+1}_{i_{k}} - x^{k}_{i_{k}} \right\rangle + \frac{c_{i_{k}}^{k}}{2} \| x^{k+1}_{i_{k}} - x^{k}_{i_{k}} \|^{2}, $$

which implies that

$$ F(x^{k+1}) \le F(x^{k}) + \left\langle \nabla_{i_{k}} f(x^{k}), x^{k+1}_{i_{k}} - x^{k}_{i_{k}} \right\rangle + \frac{c_{i_{k}}^{k}}{2} \| x^{k+1}_{i_{k}} - x^{k}_{i_{k}} \|^{2} + g_{i_{k}}(x_{i_{k}}^{k+1}) - g_{i_{k}}(x_{i_{k}}^{k}). $$

Combining with the iteration scheme (3), we have

$$ F(x^{k+1}) - F(x^{k}) \le \min_{t_{i_{k}}}\left\{ \left\langle \nabla_{i_{k}} f(x^{k}), t_{i_{k}} \right\rangle + \frac{c_{i_{k}}^{k}}{2}t_{i_{k}}^{2} + g_{i_{k}}(x_{i_{k}}^{k}+t_{i_{k}}) - g_{i_{k}}(x_{i_{k}}^{k}) \right\}, $$

where $t_{i_k}:=x_{i_k}-x^k_{i_k}$. Recall that for a given iteration point x^k, the next iteration point x^k+ 1 is obtained by using the scheme (3) where the index i_k is randomly chosen from $\{1,\dots , N\}$ with the uniform probability distribution. Conditioned on x^k and taking expectation with respect to the random index i_k, we obtain

$$ \begin{array}{@{}rcl@{}} \lefteqn{\mathbb{E}[F(x^{k+1}) - F(x^{k}) ~|~ x^{k} ] } \\ && \le \mathbb{E}\left\{\min_{t_{i_{k}}}\left\langle \nabla_{i_{k}} f(x^{k}), t_{i_{k}} \right\rangle + \frac{c_{i_{k}}^{k}}{2}t_{i_{k}}^{2} + g_{i_{k}}(x_{i_{k}}^{k}+t_{i_{k}}) - g_{i_{k}}(x_{i_{k}}^{k}) ~\Big|~ x^{k} \right\}. \end{array} $$

(29)

Now, we are going to estimate the right hand side in the above inequality

$$ \begin{array}{@{}rcl@{}} && ~~~~ \mathbb{E}\left\{\min_{t_{i_{k}}}\left\langle \nabla_{i_{k}} f(x^{k}), t_{i_{k}} \right\rangle + \frac{c_{i_{k}}^{k}}{2}t_{i_{k}}^{2} + g_{i_{k}}(x_{i_{k}}^{k}+t_{i_{k}}) - g_{i_{k}}(x_{i_{k}}^{k}) ~\Big|~ x^{k} \right\}\\ &=& \frac{1}{N}{\sum}_{i = 1}^{N} \left \{ \min_{t_{i}}\left\langle \nabla_{i} f(x^{k}), t_{i} \right\rangle + \frac{{c_{i}^{k}}}{2}{t_{i}^{2}} + g_{i}({x_{i}^{k}}+t_{i}) - g_{i}({x_{i}^{k}})\right \} \\ &\le& \frac{1}{N} \min_{t}{\sum}_{i = 1}^{N} \left\{\left\langle \nabla_{i} f(x^{k}), t_{i} \right\rangle + \frac{C}{2}{t_{i}^{2}} + g_{i}({x_{i}^{k}}+t_{i}) - g_{i}({x_{i}^{k}}) \right\}\\ &=& \frac{1}{N} \min_{y}\left\{ \left\langle \nabla f(x^{k}), y-x^{k} \right\rangle + \frac{C}{2}\|y-x^{k}\|^{2} + g(y) - g(x^{k})\right\} \\ & =& \frac{1}{N} (F_{C}(x^{k}) - F(x^{k})), \end{array} $$

(30)

where $t\!:=\!(t_1,\dots , t_N)$ and $F_{C}(x) := \min \limits _{y}\left \{ f(x) + \left \langle \nabla f(x), y - x \right \rangle + \frac {C}{2}\|y - x\|^2 + g(y) \right \}$.

Furthermore, we set

$$ \begin{array}{@{}rcl@{}} \hat{x}^{k} &:=& (I + \frac{1}{C} \partial g)^{-1} \left( x^{k} - \frac{1}{C} \nabla f(x^{k}) \right) \\ &=& \arg\min_{y}\left\{ \left\langle \nabla f(x^{k}), y-x^{k} \right\rangle + \frac{C}{2}\|y-x^{k}\|^{2} + g(y)\right\}. \end{array} $$

It follows immediately that

$$ g \left( x^{k} \right) \ge g \left( \hat{x}_{k} \right) - \left\langle \nabla f(x^{k}) + C(\hat{x}_{k} - x^{k}), x^{k} - \hat{x}_{k} \right\rangle, $$

which yields that,

$$ f \left( x^{k} \right) + g \left( x^{k} \right) \ge f \left( x^{k} \right) + \left\langle \nabla f(x^{k}), \hat{x}_{k} - x^{k} \right\rangle + \frac{C}{2}\| \hat{x}_{k} - x^{k} \|^{2} + g \left( \hat{x}_{k} \right) + \frac{C}{2}\| \hat{x}_{k} - x^{k} \|^{2}, $$

and hence

$$ F(x^{k}) \ge F_{C}(x^{k}) + \frac{C}{2}\| \hat{x}_{k} - x^{k} \|^{2}. $$

(31)

Let $\tilde {x}:= \text {Proj}_{\mathcal {X}}(x)$ for any x and thus $f(\tilde {x}) + g(\tilde {x}) = F^{*}$. Then we have

$$ \begin{array}{@{}rcl@{}} F_{C}(x) - F^{*} &=& \min_{y}\left\{ f(x) + \left\langle \nabla f(x), y-x \right\rangle + \frac{C}{2}\|y-x\|^{2} + g(y) \right\} - f(\tilde{x}) - g(\tilde{x}) \\ &\le& f(x) - f(\tilde{x}) + \left\langle \nabla f(x), \tilde{x}-x \right\rangle + \frac{C}{2}\|\tilde{x}-x\|^{2} \\ &\le& \frac{L+C}{2}\|\tilde{x}-x\|^{2} = \frac{L+C}{2}\text{dist}\left( x, {\mathcal{X}}\right)^{2}, \end{array} $$

where L is the Lipschitz constant of ∇f. Plugging x = x^k into the above inequalities, we have

$$ \begin{array}{@{}rcl@{}} F_{C}(x^{k}) - F^{*} &\le& \frac{L+C}{2}\text{dist}\left( x^{k}, {\mathcal{X}}\right)^{2} \\ &\le& \frac{\kappa(L+C)}{2} \|\hat{x}_{k}-x^{k}\|^{2} \\ &\le& \kappa(1+L/C)(F(x^{k}) - F_{C}(x^{k})), \end{array} $$

where the second inequality follows from (14) and the third inequality is a direct consequence of (31). Then we have

$$ \begin{array}{@{}rcl@{}} F(x^{k}) - F^{*} &=& F(x^{k}) - F_{C}(x^{k}) + F_{C}(x^{k}) - F^{*} \\ & \le& (1+\kappa(1+L/C))(F(x^{k}) - F_{C}(x^{k})). \end{array} $$

(32)

By (29), (30) and (32), we have

$$ \mathbb{E}[F(x^{k+1}) - F(x^{k}) ~|~ x^{k}] \leq \frac{1}{N} (F_{C}(x^{k}) - F(x^{k})) \le \frac{1}{N} \cdot \frac{1}{1+\kappa(1+L/C)}(F^{*} - F(x^{k})), $$

therefore

$$ \mathbb{E}[F(x^{k+1}) - F^{*} ~|~ x^{k}] \le \Big(1 - \frac{1}{N(1+\kappa(1+L/C))} \Big) (F(x^{k}) - F^{*}). $$

For any l ≥ 1, combining the above inequality over k = 0, 1, … , l − 1, taking expectation with all the history, we obtain

$$ \mathbb{E}[F(x^{l})-F^{*}] \le \sigma^{l} (F(x^{0})-F^{*}), $$

where $\sigma = \Big (1 - \frac {1}{N(1+\kappa (1+L/C))} \Big ) \in (0,1)$, and hence the R-BCPGM achieves a linear convergence rate in terms of the expected objective value. □