Accelerated Meta-Algorithm for Convex Optimization Problems

Abstract

An envelope called an accelerated meta-algorithm is proposed. Based on the envelope, accelerated methods for solving convex unconstrained minimization problems in various formulations can be obtained from nonaccelerated versions in a unified manner. Quasi-optimal algorithms for minimizing smooth functions with Lipschitz continuous derivatives of arbitrary order and for solving smooth minimax problems are given as applications. The proposed envelope is more general than existing ones. Moreover, better convergence estimates can be obtained in the case of this envelope and better efficiency can be achieved in practice for a number of problem formulations.

Appendices

APPENDIX 1

Below, Theorem 1 is proved relying on the proof in [14] taking into account the addition of a composite function. The following result is based on Theorem 2.1 from [14].

Theorem 3. Let ${{({{y}_{k}})}_{{k \geqslant 1}}}$ be a sequence of points in ${{\mathbb{R}}^{d}}$ and ${{({{\lambda }_{k}})}_{{k \geqslant 1}}}$ be a sequence in ${{\mathbb{R}}_{ + }}$. Define ${{({{a}_{k}})}_{{k \geqslant 1}}}$ such   that ${{\lambda }_{k}}{{A}_{k}} = a_{k}^{2}$ and ${{A}_{k}} = \sum\nolimits_{i = 1}^k \,{{a}_{i}}$. For any $k \geqslant 0$, define

$${{x}_{k}} = {{x}_{0}} - \sum\nolimits_{i = 1}^k \,{{a}_{i}}(\nabla f({{y}_{i}}) + g{\kern 1pt} '({{y}_{i}}))\;\;and\;\;{{\tilde {x}}_{k}}: = \tfrac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}{{x}_{k}} + \tfrac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}}.$$

If, for some $\sigma \in [0,1]$,

$$\left\| {{{y}_{{k + 1}}} - ({{{\tilde {x}}}_{k}} - {{\lambda }_{{k + 1}}}\nabla f({{y}_{{k + 1}}}))} \right\| \leqslant \sigma \left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|,$$

(11)

then, for any $x \in {{\mathbb{R}}^{d}}$, we have the inequalities

$$F({{y}_{k}}) - F(x) \leqslant \frac{{2{{{\left\| x \right\|}}^{2}}}}{{\mathop {\left( {\sum\limits_{i = 1}^k {\sqrt {{{\lambda }_{i}}} } } \right)}\nolimits^2 }}$$

and

$$\sum\limits_{i = 1}^k \,\frac{{{{A}_{i}}}}{{{{\lambda }_{i}}}}{{\left\| {{{y}_{i}} - {{{\tilde {x}}}_{{i - 1}}}} \right\|}^{2}} \leqslant \frac{{{{{\left\| {x\text{*}} \right\|}}^{2}}}}{{1 - {{\sigma }^{2}}}}.$$

To prove this theorem, we introduce additional lemmas based on Lemmas 2.2–2.5 and 3.1 from [14]; Lemmas 2.6 and 3.3 can be used without modifications.

Lemma 1. Given ${{\psi }_{0}}(x) = \tfrac{1}{2}{{\left\| {x - {{x}_{0}}} \right\|}^{2}}$, define ${{\psi }_{k}}(x) = {{\psi }_{{k - 1}}}(x) + {{a}_{k}}{{\Omega }_{1}}(F,{{y}_{k}},x)$ by induction. Then    ${{x}_{k}} = {{x}_{0}} - \sum\nolimits_{i = 1}^k \,{{a}_{i}}(\nabla f({{y}_{i}}) + g{\kern 1pt} '({{y}_{i}}))$ is a minimizer of the function ${{\psi }_{k}}$; moreover, ${{\psi }_{k}}(x) \leqslant {{A}_{k}}F(x) + \tfrac{1}{2}{{\left\| {x - {{x}_{0}}} \right\|}^{2}}$, where ${{A}_{k}} = \sum\nolimits_{i = 1}^k \,{{a}_{i}}$.

Lemma 2. Let ${{z}_{k}}$ be such that

$${{\psi }_{k}}({{x}_{k}}) - {{A}_{k}}F({{z}_{k}}) \geqslant 0.$$

Then, for any $x$, we have

$$F({{z}_{k}}) \leqslant F(x) + \frac{{{{{\left\| {x - {{x}_{0}}} \right\|}}^{2}}}}{{2{{A}_{k}}}}.$$

Proof. Lemma 1 implies that

$${{A}_{k}}F({{z}_{k}}) \leqslant {{\psi }_{k}}({{x}_{k}}) \leqslant {{\psi }_{k}}(x) \leqslant {{A}_{k}}F(x) + \frac{1}{2}{{\left\| {x - {{x}_{0}}} \right\|}^{2}}.$$

Lemma 3. For any $x$, it is true that

$$\begin{gathered} {{\psi }_{{k + 1}}}(x) - {{A}_{{k + 1}}}F({{y}_{{k + 1}}}) - ({{\psi }_{k}}({{x}_{k}}) - {{A}_{k}}F({{z}_{k}})) \\ \geqslant {{A}_{{k + 1}}}(\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))\left( {\frac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}x + \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{z}_{k}} - {{y}_{{k + 1}}}} \right) + \frac{1}{2}{{\left\| {x - {{x}_{k}}} \right\|}^{2}}. \\ \end{gathered} $$

Proof. First, simple computations yield

$${{\psi }_{k}}(x) = {{\psi }_{k}}({{x}_{k}}) + \frac{1}{2}{{\left\| {x - {{x}_{k}}} \right\|}^{2}}$$

and

$${{\psi }_{{k + 1}}}(x) = {{\psi }_{k}}({{x}_{k}}) + \frac{1}{2}{{\left\| {x - {{x}_{k}}} \right\|}^{2}} + {{a}_{{k + 1}}}{{\Omega }_{1}}(f,{{y}_{{k + 1}}},x);$$

thus,

$${{\psi }_{{k + 1}}}(x) - {{\psi }_{k}}({{x}_{k}}) = {{a}_{{k + 1}}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},x) + \frac{1}{2}{{\left\| {x - {{x}_{k}}} \right\|}^{2}}.$$

(12)

Now we want ${{A}_{{k + 1}}}F({{z}_{{k + 1}}}) - {{A}_{k}}F({{z}_{k}})$ to be a lower bound for inequality (12) when we compute $x = {{x}_{{k + 1}}}$. Using ${{\Omega }_{1}}(F,{{y}_{{k + 1}}},{{z}_{k}}) \leqslant f({{z}_{k}})$ yields

$$\begin{gathered} {{a}_{{k + 1}}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},x) = {{A}_{{k + 1}}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},x) - {{A}_{k}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},x) = {{A}_{{k + 1}}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},x) - {{A}_{k}}\nabla F({{y}_{{k + 1}}})(x - {{z}_{k}}) \\ - \;{{A}_{k}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},{{z}_{k}}) = {{A}_{{k + 1}}}{{\Omega }_{1}}\left( {F,{{y}_{{k + 1}}},x - \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}(x - {{z}_{k}})} \right) - {{A}_{k}}{{\Omega }_{1}}(F,{{y}_{{k + 1}}},{{z}_{k}}) \geqslant {{A}_{{k + 1}}}F({{y}_{{k + 1}}}) - {{A}_{k}}F({{z}_{k}}) \\ \, + {{A}_{{k + 1}}}(\nabla f({{y}_{{k + 1}}}) + g'({{y}_{{k + 1}}}))\left( {\frac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}x + \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{z}_{k}} - {{y}_{{k + 1}}}} \right), \\ \end{gathered} $$

which completes the proof.

Lemma 4. Let ${{\lambda }_{{k + 1}}}: = \tfrac{{a_{{k + 1}}^{2}}}{{{{A}_{{k + 1}}}}}$ and $\mathop {\tilde {x}}\nolimits_k : = \tfrac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}{{x}_{k}} + \tfrac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}}$. Then

$$\begin{gathered} {{\psi }_{{k + 1}}}({{x}_{{k + 1}}}) - {{A}_{{k + 1}}}F({{y}_{{k + 1}}}) - ({{\psi }_{k}}({{x}_{k}}) - {{A}_{k}}F({{y}_{k}})) \\ \geqslant \frac{{{{A}_{{k + 1}}}}}{{2{{\lambda }_{{k + 1}}}}}({{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}^{2}} - {{\left\| {{{y}_{{k + 1}}} - ({{{\tilde {x}}}_{k}} - {{\lambda }_{{k + 1}}}(\nabla f({{y}_{{k + 1}}})) + g{\kern 1pt} '({{y}_{{k + 1}}}))} \right\|}^{2}}). \\ \end{gathered} $$

Additionally applying inequality (11) yields

$${{\psi }_{k}}({{x}_{k}}) - {{A}_{k}}F({{y}_{k}}) \geqslant \frac{{1 - {{\sigma }^{2}}}}{2}\sum\limits_{i = 1}^k \,\frac{{{{A}_{i}}}}{{{{\lambda }_{i}}}}{{\left\| {{{y}_{i}} - {{{\tilde {x}}}_{{i - 1}}}} \right\|}^{2}}.$$

Proof. Using Lemma 3 with ${{z}_{k}} = {{y}_{k}}$ and $x = {{x}_{{k + 1}}}$, we obtain (for $\tilde {x}: = \tfrac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}x + \tfrac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}}$)

$$\begin{gathered} (\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))\left( {\frac{{{{a}_{{k + 1}}}}}{{{{A}_{{k + 1}}}}}x + \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}} - {{y}_{{k + 1}}}} \right) + \frac{1}{{2{{A}_{{k + 1}}}}}{{\left\| {x - {{x}_{k}}} \right\|}^{2}} = (\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))(\tilde {x} - {{y}_{{k + 1}}}) \\ + \;\frac{1}{{2{{A}_{{k + 1}}}}}{{\left\| {\frac{{{{A}_{{k + 1}}}}}{{{{a}_{{k + 1}}}}}\left( {\tilde {x} - \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}}} \right) - {{x}_{k}}} \right\|}^{2}} = (\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))(\tilde {x} - {{y}_{{k + 1}}}) + \frac{{{{A}_{{k + 1}}}}}{{2a_{{k + 1}}^{2}}}{{\left\| {\tilde {x} - \left( {\frac{{{{a}_{{k + 1}}}}}{{{{A}_{k}}}}{{x}_{k}} + \frac{{{{A}_{k}}}}{{{{A}_{{k + 1}}}}}{{y}_{k}}} \right)} \right\|}^{2}}, \\ \end{gathered} $$

whence

$$\begin{gathered} {{\psi }_{{k + 1}}}({{x}_{{k + 1}}}) - {{A}_{{k + 1}}}F({{y}_{{k + 1}}}) - ({{\psi }_{k}}({{x}_{k}}) - {{A}_{k}}F({{y}_{k}})) \\ \geqslant {{A}_{{k + 1}}}\mathop {\min}\limits_{x \in {{\mathbb{R}}^{d}}} \left\{ {(\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))(x - {{y}_{{k + 1}}}) + \frac{1}{{2{{\lambda }_{{k + 1}}}}}{{{\left\| {x - {{{\tilde {x}}}_{k}}} \right\|}}^{2}}} \right\}. \\ \end{gathered} $$

The value of the minimum is easy to compute.

The first inequality in Theorem 3 is proved by combining Lemmas 4 and 2 with Lemma 2.5 from [14]. The second inequality in Theorem 3 follows from Lemmas 4 and 1.

The following lemma proves that the minimization of the Taylor series of order $p$ for (4) can be represented as an implicit gradient step for a large stepsize.

Lemma 5. Inequality (11) holds at $\sigma = 1{\text{/}}2$ for (4), which implies that

$$\frac{1}{2} \leqslant {{\lambda }_{{k + 1}}}\frac{{{{L}_{p}}{{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}}}}{{(p - 1)!}} \leqslant \frac{p}{{p + 1}}.$$

(13)

Proof. The optimality condition yields

$${{\nabla }_{y}}{{f}_{p}}({{y}_{{k + 1}}},{{\tilde {x}}_{k}}) + \frac{{{{L}_{p}}(p + 1)}}{{p!}}({{y}_{{k + 1}}} - {{\tilde {x}}_{k}}){{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}^{{p - 1}}} + g{\kern 1pt} '({{y}_{{k + 1}}}) = 0.$$

(14)

It follows that

$$\begin{gathered} {{y}_{{k + 1}}} - ({{{\tilde {x}}}_{k}} - {{\lambda }_{{k + 1}}}(\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}}))) = {{\lambda }_{{k + 1}}}(\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}})) \\ \, - \frac{{p!}}{{{{L}_{p}}(p + 1){{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}}}}({{\nabla }_{y}}{{f}_{p}}({{y}_{{k + 1}}},{{{\tilde {x}}}_{k}}) + g{\kern 1pt} '({{y}_{{k + 1}}})). \\ \end{gathered} $$

Using the Taylor series, we find that the gradient of the function satisfies

$$\left\| {\nabla f(y) - {{\nabla }_{y}}{{f}_{p}}(y,x)} \right\| \leqslant \frac{{{{L}_{p}}}}{{p!}}{{\left\| {y - x} \right\|}^{p}};$$

thus,

$$\begin{gathered} \left\| {{{y}_{{k + 1}}} - ({{{\tilde {x}}}_{k}} - {{\lambda }_{{k + 1}}}(\nabla f({{y}_{{k + 1}}}) + g{\kern 1pt} '({{y}_{{k + 1}}})))} \right\| \leqslant {{\lambda }_{{k + 1}}}\frac{{{{L}_{p}}}}{{p!}}{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}^{p}} \\ + \;\left| {{{\lambda }_{{k + 1}}} - \frac{{p!}}{{{{L}_{p}}(p + 1){{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}}}}} \right|\left\| {{{\nabla }_{y}}{{f}_{p}}({{y}_{{k + 1}}},{{{\tilde {x}}}_{k}}) + g{\kern 1pt} '({{y}_{{k + 1}}})} \right\| \\ \leqslant \left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|\left( {{{\lambda }_{{k + 1}}}\frac{{{{L}_{p}}}}{{p!}}{{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}} + \left| {{{\lambda }_{{k + 1}}}\frac{{{{L}_{p}}(p + 1){{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}}}}{{p!}} - 1{\kern 1pt} } \right|} \right) \\ \, = \left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|\left( {\frac{\eta }{p} + \left| {\eta \frac{{p + 1}}{p} - 1} \right|} \right), \\ \end{gathered} $$

where we used (14) in the second inequality and set $\eta : = {{\lambda }_{{k + 1}}}\tfrac{{{{L}_{p}}{{{\left\| {{{y}_{{k + 1}}} - {{{\tilde {x}}}_{k}}} \right\|}}^{{p - 1}}}}}{{(p - 1)!}}$ in the last equality. The final result is obtained by assuming that $1{\text{/}}2 \leqslant \eta \leqslant p{\text{/}}(p + 1)$ in (13).

Finally, replacing $\left\| {x\text{*}} \right\|$ by $\left\| {{{x}_{0}} - x\text{*}} \right\|$ in Lemma 3.3 and applying Lemma 3.4 from [14], we complete the proof of Theorem 1.

APPENDIX 2

Proof of Theorem 2. Since $F$ is an $r$-uniformly convex function, we obtain

$${{R}_{{k + 1}}} = \left\| {{{z}_{{k + 1}}} - {{x}_{{{*}}}}} \right\| \leqslant {{\left( {\frac{{r(F({{z}_{{k + 1}}}) - F({{x}_{{{*}}}}))}}{{{{\sigma }_{r}}}}} \right)}^{{1/r}}}\;\mathop \leqslant \limits^{(5)} \;{{\left( {\frac{{r\left( {\frac{{{{c}_{p}}{{L}_{p}}R_{k}^{{p + 1}}}}{{N_{k}^{{\tfrac{{3p + 1}}{2}}}}}} \right)}}{{{{\sigma }_{r}}}}} \right)}^{{1/r}}} = {{\left( {\frac{{r{{c}_{p}}{{L}_{p}}R_{k}^{{p + 1}}}}{{{{\sigma }_{r}}N_{k}^{{\tfrac{{3p + 1}}{2}}}}}} \right)}^{{1/r}}}\;\mathop \leqslant \limits^{(9)} \;{{\left( {\frac{{R_{k}^{{p + 1}}}}{{{{2}^{r}}R_{k}^{{p + 1 - r}}}}} \right)}^{{1/r}}} = \frac{{{{R}_{k}}}}{2}.$$

Now the total number of steps in method 1 can be estimated as follows:

$$\begin{gathered} \sum\limits_{k = 0}^K \,{{N}_{k}} \leqslant \sum\limits_{k = 0}^K \,{{\left( {\frac{{r{{c}_{p}}{{L}_{p}}{{2}^{r}}}}{{{{\sigma }_{r}}}}R_{k}^{{p + 1 - r}}} \right)}^{{\tfrac{2}{{3p + 1}}}}} + K = \sum\limits_{k = 0}^K \,{{\left( {\frac{{r{{c}_{p}}{{L}_{p}}{{2}^{r}}}}{{{{\sigma }_{r}}}}{{{({{R}_{0}}{{2}^{{ - k}}})}}^{{p + 1 - r}}}} \right)}^{{\tfrac{2}{{3p + 1}}}}} + K \\ \, = {{\left( {\frac{{r{{c}_{p}}{{L}_{p}}{{2}^{r}}R_{0}^{{p + 1 - r}}}}{{{{\sigma }_{r}}}}} \right)}^{{\tfrac{2}{{3p + 1}}}}}\sum\limits_{k = 0}^K \,{{2}^{{\tfrac{{ - 2(p + 1 - r)k}}{{3p + 1}}}}} + K. \\ \end{gathered} $$