Fast gradient methods for uniformly convex and weakly smooth problems

Abstract

In this paper, acceleration of gradient methods for convex optimization problems with weak levels of convexity and smoothness is considered. Starting from the universal fast gradient method which was designed to be an optimal method for weakly smooth problems whose gradients are Hölder continuous, its momentum is modified appropriately so that it can also accommodate uniformly convex and weakly smooth problems. Different from the existing works, fast gradient methods proposed in this paper do not use the restarting technique but use momentums that are suitably designed to reflect both the uniform convexity and weak smoothness information of the target energy function. Both theoretical and numerical results that support the superiority of the proposed methods are presented.

Appendices

Appendix A. Recurrence inequalities

This appendix presents several useful recurrence inequalities used throughout this paper and their proofs. Motivated by [20, Lemma 8] and [7, Theorem 3], we state the following useful lemma.

Lemma A.1

Let {A_n}_n≥ 1 be an increasing sequence of positive real numbers that satisfies

$$ (A_{n+1} - A_{n})^{\gamma} \geq C A_{n}A_{n+1}^{\gamma - 1}, \quad n \geq1 $$

for some 1 ≤ γ ≤ 2 and C > 0. Then we have

$$ A_{n} \geq A_{1} \left( 1+ \frac{C^{1/\gamma}}{2} \right)^{\gamma(n-1)}, \quad n \geq 1. $$

Proof

Take any n ≥ 1. Since 1/2 ≤ 1/γ ≤ 1, the following inequality holds:

$$ A_{n+1} - A_{n} \leq (A_{n+1}^{1- 1/\gamma} + A_{n}^{1- 1/\gamma} ) (A_{n+1}^{1/\gamma} - A_{n}^{1/\gamma} ) . $$

It follows that

$$ C A_{n} A_{n+1}^{\gamma - 1} \leq (A_{n+1}^{1- 1/\gamma} + A_{n}^{1- 1/\gamma} )^{\gamma} (A_{n+1}^{1/\gamma} - A_{n}^{1/\gamma} )^{\gamma} \leq 2^{\gamma} A_{n+1}^{\gamma - 1} (A_{n+1}^{1/\gamma} - A_{n}^{1/\gamma} )^{\gamma}. $$

Now, we have

$$ A_{n+1}^{1/\gamma} - A_{n}^{1/\gamma} \geq \frac{C^{1/\gamma}}{2} A_{n}^{1/\gamma} \quad \Leftrightarrow \quad A_{n+1} \geq \left( 1 + \frac{C^{1/\gamma}}{2} \right)^{\gamma} A_{n}. $$

We get the desired result by applying the above inequality recursively. □

The next lemma is useful when we prove sublinear convergence rate of some fast gradient methods. We note that the proof of Lemma A.2 closely follows [15, Lemma 1].

Lemma A.2

Let {A_n}_n≥ 1 be an increasing sequence of positive real numbers that satisfies

$$ A_{n+1} - A_{n} \geq C A_{n}^{\gamma}, \quad n \geq 1 $$

for some 0 ≤ γ < 1 and C > 0. Then we have

$$ A_{n} \geq \min \left\{ A_{1}, \left( \frac{C}{2 (2^{\frac{1}{1-\gamma}} - 1 )} \right)^{\frac{1}{1-\gamma}} \right\} n^{\frac{1}{1-\gamma}}, \quad n \geq 1. $$

Proof

Take any n ≥ 1. Since A_n+ 1 ≥ A_n, we get

$$ A_{n+1} - A_{n} \geq CA_{n}^{\gamma} \geq C A_{n} A_{n+1}^{-(1-\gamma)}, $$

or equivalently,

$$ \frac{A_{n+1}}{A_{n}} \geq 1 + C A_{n+1}^{-(1-\gamma)}. $$

Writing $A_{n} = B_{n} n^{\frac {1}{1- \gamma }}$, we have

$$ \frac{B_{n+1}}{B_{n}} \geq \left( \frac{n}{n+1} \right)^{\frac{1}{1-\gamma}} \left( 1 + \frac{C B_{n+1}^{-(1 - \gamma)}}{n+1} \right). $$

(A.1)

The right-hand side of (A.1) is greater than or equal to 1 if and only if

$$ B_{n+1} \leq \left[\frac{n+1}{C} \left( \left( 1+ \frac{1}{n} \right)^{\frac{1}{1-\gamma}} - 1 \right) \right]^{-\frac{1}{1-\gamma}} $$

(A.2)

From the fact that the right-hand side of (A.2) increases as n increases, we deduce that a sufficient condition to satisfy B_n+ 1 ≥ B_n is that

$$ B_{n+1} \leq \left( \frac{C}{2(2^{\frac{1}{1-\gamma}} - 1 )} \right)^{\frac{1}{1-\gamma}}. $$

Then it is straightforward by mathematical induction that

$$ B_{n} \geq \min \left\{ B_{1}, \left( \frac{C}{2(2^{\frac{1}{1-\gamma}} - 1 )} \right)^{\frac{1}{1-\gamma}} \right\}, \quad n \geq 1, $$

which implies the desired result. □

Appendix B. Numerical verification of Claim 4.8

This appendix is devoted to discussions on Claim 4.8. We prove a special case of Claim 4.8 and then present numerical evidences for the other cases. First, we consider the situation when p = 2, i.e., the function f in (1.5) is strongly convex.

Proposition B.1

Claim 4.8 holds when p = 2.

Proof

If p = 2, then (4.1) and (4.8) reduces to (3.6) and (3.7), respectively. Therefore, the desired result can be obtained by the same argument as Theorem 3.8. □

Proposition B.1 means that Claim 4.8 is indeed a generalization of Theorem 3.8 to the case p ≥ 2. In the remainder of this appendix, we assume that p > 2 ≥ q ≥ 1. We show that the following claim is a sufficient condition to ensure Claim 4.8.

Claim B.2

Let {A_n}_n≥ 1 be an increasing sequence of positive real numbers that satisfies

$$ (A_{n+1} - A_{n})^{2} \geq C A_{n}^{1 - \frac{2-q}{q} \left( 1 + \frac{2(p-q)}{p(3q-2)} \right)} \sum\limits_{j=1}^{n} \frac{(A_{j} - A_{j-1})^{\frac{2}{p}}}{A_{j}^{\frac{p-2}{p} \frac{2(p-q)}{p(3q-2)}}}, \quad n \geq 1 $$

(B.1)

for some p > 2 ≥ q ≥ 1 and C > 0. Then we have

$$ A_{n} \geq \widetilde{C} n^{\frac{p(3q-2)}{2(p-q)}}, \quad n \geq 1, $$

where $\widetilde {C}$ is a positive constant depending on p, q, C, and A₁ only.

Remark B.3

Replacing A_n, A_n+ 1 − A_n, and ${\sum }_{j=1}^{n} \cdot$ in (B.1) by y(t), $y^{\prime }(t)$, and ${{\int \limits }_{0}^{t}} \cdot ds$, respectively, one can obtain the ordinary differential equation

$$ y^{\prime}(t)^{2} = C y(t)^{1 - \frac{2-q}{q} \left( 1 + \frac{2(p-q)}{p(3q-2)} \right)} {{\int}_{0}^{t}} \frac{y^{\prime}(s)^{\frac{2}{p}}}{y(s)^{\frac{p-2}{p} \frac{2(p-q)}{p(3q-2)}}} ds, $$

(B.2)

which is a continuous analogue of (B.1). If we impose the initial condition y(0) = 0 to (B.2), then we can readily verify that (B.2) admits a solution

$$ y(t) = \widehat{C} t^{\frac{p(3q-2)}{2(p-q)}}, \quad t \geq 0, $$

where $\widehat {C}$ is an appropriate constant depending on p, q, and C. That is, the solution of (B.2) has the same growth rate as the conclusion of Claim B.2.

Even though Claim B.2 has the rather complex structure, it is in fact a generalization of (A.2). If we set p = 2 and 1 ≤ q < 2 in (B.1), then we get

$$ (A_{n+1} - A_{n}) \geq C^{\frac{1}{2}} A_{n}^{\frac{4q-4}{3q-2}}, \quad n \geq 1, $$

which has the same form as (A.2) (note that $0 \leq \frac {4q-4}{3q-2} < 1$ if 1 ≤ q < 2).

Proposition B.4

Assume that p > 2 ≥ q ≥ 1. Claim B.2 implies Claim 4.8.

Proof

The starting point of the proof is (4.1); (4.4) and (4.8) imply that

$$ \begin{array}{ll} (A_{n+1} - A_{n})^{2} &\geq \frac{A_{n+1} {\sum}_{j=1}^{n} \delta_{j-1}^{\frac{p-2}{p}} \mu^{\frac{2}{p}} (A_{j} - A_{j-1}) }{L_{n+1}} \\ &\geq \frac{C_{\delta}^{\frac{p-2}{p}}C_{\epsilon}^{\frac{2-q}{q}}}{2 \kappa} A_{n+1}^{1 - \frac{2-q}{q} \left( 1 + \frac{2(p-q)}{p(3q-2)} \right)} \sum\limits_{j=1}^{n} \frac{(A_{j} - A_{j-1})^{\frac{2}{p}}}{A_{j}^{\frac{p-2}{p} \frac{2(p-q)}{p(3q-2)}}} \\ &\geq \frac{C_{\delta}^{\frac{p-2}{p}}C_{\epsilon}^{\frac{2-q}{q}}}{2 \kappa} A_{n}^{1 - \frac{2-q}{q} \left( 1 + \frac{2(p-q)}{p(3q-2)} \right)} \sum\limits_{j=1}^{n} \frac{(A_{j} - A_{j-1})^{\frac{2}{p}}}{A_{j}^{\frac{p-2}{p} \frac{2(p-q)}{p(3q-2)}}} , \end{array} $$

where κ was defined in (2.3). Similarly to (3.8), one can verify that A₁ has a lower bound depending on p, q, and L only. Hence, if we assume that Claim B.2 holds, then we can conclude that $A_{n} \geq \widetilde {C}n^{\frac {p(3q-2)}{2(p-q)}}\!,$ where $\widetilde {C}$ is a positive constant depending on p, q, L, μ, C_𝜖, and C_δ. It is straightforward to prove that

$$ F(x_{n}) - F(x^{*}) \leq \frac{1}{2 \widetilde{C}n^{\frac{p(3q-2)}{2(p-q)}}} \left( \|x_{0} - x^{*} \|^{2} + \frac{C}{\widetilde{C}^{\frac{2(p-q)}{p(3q-2)}}} \left( 1 + \log n \right) \right) $$

by invoking Theorem 4.4 and using the same argument as Theorem 3.8. □

We verify Claim B.2 by numerical experiments. Figure 7 plots A_n and $n^{\frac {p(3q-2)}{2(p-q)}}$ with respect to n in log-log scale, where A_n is generated by the recurrence relation

$$ A_{1} =1, \quad (A_{n+1} - A_{n})^{2} = A_{n}^{1 - \frac{2-q}{q} \left( 1 + \frac{2(p-q)}{p(3q-2)} \right)} \sum\limits_{j=1}^{n} \frac{(A_{j} - A_{j-1})^{\frac{2}{p}}}{A_{j}^{\frac{p-2}{p} \frac{2(p-q)}{p(3q-2)}}}, n \geq 1 $$

(B.3)

for various choices of p and q. In all cases, the slope of the graph for A_n is a bit greater than that of the graph for $n^{\frac {p(3q-2)}{2(p-q)}}$. That is, the asymptotic growth rate of A_n is observed to be greater than $n^{\frac {p(3q-2)}{2(p-q)}}$, which implies Claim B.2. Thanks to Proposition B.4, the validity of Claim 4.8 is ensured by the above numerical results.

Fig. 7

Growth of A_n generated by the recurrence relation (B.3) and $n^{\frac {p(3q-2)}{2(p-q)}}$ for various p and q such that p > 2 ≥ q ≥ 1