First-order optimization algorithms via inertial systems with Hessian driven damping

Abstract

In a Hilbert space setting, for convex optimization, we analyze the convergence rate of a class of first-order algorithms involving inertial features. They can be interpreted as discrete time versions of inertial dynamics involving both viscous and Hessian-driven dampings. The geometrical damping driven by the Hessian intervenes in the dynamics in the form $\nabla ^2 f (x(t)) \dot{x} (t)$. By treating this term as the time derivative of $\nabla f (x (t))$, this gives, in discretized form, first-order algorithms in time and space. In addition to the convergence properties attached to Nesterov-type accelerated gradient methods, the algorithms thus obtained are new and show a rapid convergence towards zero of the gradients. On the basis of a regularization technique using the Moreau envelope, we extend these methods to non-smooth convex functions with extended real values. The introduction of time scale factors makes it possible to further accelerate these algorithms. We also report numerical results on structured problems to support our theoretical findings.

Auxiliary results

1.1 Extended descent lemma

Lemma 1

Let $f: {{\mathcal {H}}}\rightarrow {\mathbb R}$ be a convex function whose gradient is L-Lipschitz continuous. Let $s \in ]0,1/L]$. Then for all $(x,y) \in {{\mathcal {H}}}^2$, we have

$$\begin{aligned} f(y - s \nabla f (y)) \le f (x) + \left\langle \nabla f (y), y-x \right\rangle -\frac{s}{2} \Vert \nabla f (y) \Vert ^2 -\frac{s}{2} \Vert \nabla f (x)- \nabla f (y) \Vert ^2 .\nonumber \\ \end{aligned}$$

(28)

Proof

Denote $y^+=y - s \nabla f (y)$. By the standard descent lemma applied to $y^+$ and y, and since $sL \le 1$ we have

$$\begin{aligned} f(y^+) \le f(y) - \frac{s}{2}\left( {2-Ls}\right) \Vert \nabla f (y) \Vert ^2 \le f(y) - \frac{s}{2} \Vert \nabla f (y) \Vert ^2. \end{aligned}$$

(29)

We now argue by duality between strong convexity and Lipschitz continuity of the gradient of a convex function. Indeed, using Fenchel identity, we have

$$\begin{aligned} f(y) = \langle \nabla f(y),\,y \rangle - f^*(\nabla f(y)) . \end{aligned}$$

L-Lipschitz continuity of the gradient of f is equivalent to 1/L-strong convexity of its conjugate $f^*$. This together with the fact that $(\nabla f)^{-1}=\partial f^*$ gives for all $(x,y) \in {{\mathcal {H}}}^2$,

$$\begin{aligned} f^*(\nabla f(y)) \ge f^*(\nabla f(x)) + \langle x,\,\nabla f(y)-\nabla f(x) \rangle + \frac{1}{2L}\left\| {\nabla f(x)-\nabla f(y)}\right\| ^2 . \end{aligned}$$

Inserting this inequality into the Fenchel identity above yields

$$\begin{aligned} f(y)&\le - f^*(\nabla f(x)) + \langle \nabla f(y),\,y \rangle - \langle x,\,\nabla f(y)-\nabla f(x) \rangle - \frac{1}{2L}\left\| {\nabla f(x)-\nabla f(y)}\right\| ^2 \\&= - f^*(\nabla f(x)) + \langle x,\,\nabla f(x) \rangle + \langle \nabla f(y),\,y-x \rangle - \frac{1}{2L}\left\| {\nabla f(x)-\nabla f(y)}\right\| ^2 \\&= f(x) + \langle \nabla f(y),\,y-x \rangle - \frac{1}{2L}\left\| {\nabla f(x)-\nabla f(y)}\right\| ^2 \\&\le f(x) + \langle \nabla f(y),\,y-x \rangle - \frac{s}{2}\left\| {\nabla f(x)-\nabla f(y)}\right\| ^2 . \end{aligned}$$

Inserting the last bound into (29) completes the proof. $\square$

1.2 Proof of (27)

Proof

We have

$$\begin{aligned} {{\,\mathrm{prox}\,}}_{f}^{M}(x)&= {{\,\mathrm{argmin}\,}}_{z \in {\mathbb R}^n} \frac{1}{2}\left\| {z - x}\right\| _M^2 + f(z) \\&= {{\,\mathrm{argmin}\,}}_{z \in {\mathbb R}^n} \frac{1}{2s}\left\| {z - x}\right\| ^2 - \frac{1}{2}\left\| {A(z - x)}\right\| ^2 + \frac{1}{2}\left\| {y-A z}\right\| ^2 + g(z) . \end{aligned}$$

By the Pythagoras relation, we then get

$$\begin{aligned} {{\,\mathrm{prox}\,}}_{f}^M(x)&= {{\,\mathrm{argmin}\,}}_{z \in {\mathbb R}^n} \frac{1}{2s}\left\| {z - x}\right\| ^2 + \frac{1}{2}\left\| {y-A x}\right\| ^2 - \langle A(x-z),\,A x - y \rangle + g(z) \\&= {{\,\mathrm{argmin}\,}}_{z \in {\mathbb R}^n} \frac{1}{2s}\left\| {z - x}\right\| ^2 - \langle z - x,\,A^*\left( {y - A x}\right) \rangle + g(z) \\&= {{\,\mathrm{argmin}\,}}_{z \in {\mathbb R}^n} \frac{1}{2s}\left\| {z - \left( {x - s A^*\left( {A x - y}\right) }\right) }\right\| ^2 + g(z) \\&= {{\,\mathrm{prox}\,}}_{s g}\left( {x - s A^*\left( {A x - y}\right) }\right) . \end{aligned}$$

$\square$

1.3 Closed-form solutions of $\text {(DIN-AVD)}_{\alpha ,\beta ,b}\,$ for quadratic functions

We here provide the closed form solutions to $\text {(DIN-AVD)}_{\alpha ,\beta ,b}\,$ for the quadratic objective $f: {\mathbb R}^n \rightarrow \langle Ax,\,x \rangle$, where A is a symmetric positive definite matrix. The case of a semidefinite positive matrix A can be treated similarly by restricting the analysis to $\ker (A)^\top$. Projecting $\text {(DIN-AVD)}_{\alpha ,\beta ,b}\,$ on the eigenspace of A, one has to solve n independent one-dimensional ODEs of the form

$$\begin{aligned} \ddot{x}_i(t) + \left( {\frac{\alpha }{t}+\beta (t)\lambda _i}\right) \dot{x}_i(t) + \lambda _i b(t) x_i(t) = 0, \qquad i=1,\ldots ,n . \end{aligned}$$

where $\lambda _i > 0$ is an eigenvalue of A. In the following, we drop the subscript i.

Case $\varvec{\beta (t) \equiv \beta , b(t)=b+\gamma /t, \beta \ge 0, b > 0, \gamma \ge 0}$: The ODE reads

$$\begin{aligned} \ddot{x}(t) + \left( {\frac{\alpha }{t}+\beta \lambda }\right) \dot{x}(t) + \lambda \left( {b+\frac{\gamma }{t}}\right) x(t) = 0 . \end{aligned}$$

(30)

• If $\beta ^2\lambda ^2 \ne 4b\lambda$: set

  $$\begin{aligned} \xi = \sqrt{\beta ^2\lambda ^2 - 4b\lambda }, \, \kappa = \lambda \frac{\gamma -\alpha \beta /2}{\xi }, \, \sigma = (\alpha -1)/2 . \end{aligned}$$

  Using the relationship between the Whitaker functions and the Kummer’s confluent hypergeometric functions M and U, see [16], the solution to (30) can be shown to take the form

  $$\begin{aligned} x(t) = \xi ^{\alpha /2} e^{-(\beta \lambda +\xi )t/2}\left[ {c_1 M(\alpha /2-\kappa ,\alpha ,\xi t) + c_2 U(\alpha /2-\kappa ,\alpha ,\xi t)}\right] , \end{aligned}$$

  where $c_1$ and $c_2$ are constants given by the initial conditions.

• If $\beta ^2\lambda ^2 = 4b\lambda$: set $\zeta =2\sqrt{\lambda \left( {\gamma -\alpha \beta /2}\right) }$. The solution to (30) takes the form

  $$\begin{aligned} x(t) = t^{-\left( {\alpha -1}\right) /2}e^{-\beta \lambda t/2}\left[ {c_1 J_{(\alpha -1)/2}(\zeta \sqrt{t}) + c_2 Y_{(\alpha -1)/2}(\zeta \sqrt{t})}\right] , \end{aligned}$$

  where $J_\nu$ and $Y_\nu$ are the Bessel functions of the first and second kind.

When $\beta > 0$, one can clearly see the exponential decrease forced by the Hessian. From the asymptotic expansions of M, U, $J_{\nu }$ and $Y_{\nu }$ for large t, straightforward computations provide the behaviour of |x(t)| for large t as follows:

• If $\beta ^2\lambda ^2 > 4b\lambda$, we have

  $$\begin{aligned} |x(t)| = {{\mathcal {O}}}\left( {t^{-\frac{\alpha }{2}+|\kappa |} e^{-\frac{\beta \lambda -\xi }{2}t}}\right) = {{\mathcal {O}}}\left( {e^{-\frac{2b}{\beta }t - \left( {\frac{\alpha }{2}-|\kappa |}\right) \log (t)}}\right) . \end{aligned}$$

• If $\beta ^2\lambda ^2 < 4b\lambda$, whence $\xi \in i {\mathbb R}^+_*$ and $\kappa \in i {\mathbb R}$, we have

  $$\begin{aligned} |x(t)| = {{\mathcal {O}}}\left( {t^{-\frac{\alpha }{2}} e^{-\frac{\beta \lambda }{2}t}}\right) . \end{aligned}$$

• If $\beta ^2\lambda ^2 = 4b\lambda$, we have

  $$\begin{aligned} |x(t)| = {{\mathcal {O}}}\left( {t^{-\frac{2\alpha -1}{4}} e^{-\frac{\beta \lambda }{2}t}}\right) . \end{aligned}$$

Case $\varvec{\beta (t) = t^{\beta }, b(t)=ct^{\beta -1}, \beta \ge 0, c > 0}$: The ODE reads now

$$\begin{aligned} \ddot{x}(t) + \left( {\frac{\alpha }{t}+t^\beta \lambda }\right) \dot{x}(t) + c\lambda t^{\beta -1} x(t) = 0 . \end{aligned}$$

Let us make the change of variable $t :=\tau ^{\frac{1}{\beta +1}}$. Let $y(\tau ) :=x\left( {\tau ^{\frac{1}{\beta +1}}}\right)$. By the standard derivation chain rule, it is straightforward to show that y obeys the ODE

$$\begin{aligned} \ddot{y}(\tau ) + \left( {\frac{\alpha +\beta }{(1+\beta )\tau }+\frac{\lambda }{1+\beta }}\right) \dot{y}(\tau ) + \frac{c\lambda }{(1+\beta )^2\tau } y(\tau ) = 0 . \end{aligned}$$

It is clear that this is a special case of (30). Since $\beta$ and $\lambda > 0$, set

$$\begin{aligned} \xi = \frac{\lambda }{1+\beta }, \, \kappa = -\frac{\alpha +\beta -c}{1+\beta }, \, \sigma = \frac{\alpha +\beta }{2(1+\beta )} - \frac{1}{2} . \end{aligned}$$

It follows from the first case above that

$$\begin{aligned} x(t) = \xi ^{\sigma +1/2} e^{-\frac{\lambda \tau }{1+\beta }}\left[ {c_1 M\left( {\sigma -\kappa +1/2,\frac{\alpha +\beta }{1+\beta },\xi \tau }\right) + c_2 U\left( {\sigma -\kappa +1/2,\frac{\alpha +\beta }{1+\beta },\xi \tau }\right) }\right] . \end{aligned}$$

Asymptotic estimates can also be derived similarly to above. We omit the details for the sake of brevity.