Preconditioned Accelerated Gradient Descent Methods for Locally Lipschitz Smooth Objectives with Applications to the Solution of Nonlinear PDEs

Abstract

We develop a theoretical foundation for the application of Nesterov’s accelerated gradient descent method (AGD) to the approximation of solutions of a wide class of partial differential equations (PDEs). This is achieved by proving the existence of an invariant set and exponential convergence rates when its preconditioned version (PAGD) is applied to minimize locally Lipschitz smooth, strongly convex objective functionals. We introduce a second-order ordinary differential equation (ODE) with a preconditioner built-in and show that PAGD is an explicit time-discretization of this ODE, which requires a natural time step restriction for energy stability. At the continuous time level, we show an exponential convergence of the ODE solution to its steady state using a simple energy argument. At the discrete level, assuming the aforementioned step size restriction, the existence of an invariant set is proved and a matching exponential rate of convergence of the PAGD scheme is derived by mimicking the energy argument and the convergence at the continuous level. Applications of the PAGD method to numerical PDEs are demonstrated with certain nonlinear elliptic PDEs using pseudo-spectral methods for spatial discretization, and several numerical experiments are conducted. The results confirm the global geometric and mesh size-independent convergence of the PAGD method, with an accelerated rate that is improved over the preconditioned gradient descent (PGD) method.

Appendices

Appendix 1: An IVP as the Limit of the PAGD Method

1.1 A.1. Derivation of the ODE

Let us start with the same approach as in [33]. We assume, as an ansatz, that PAGD is a discretization of an ODE, which has a solution $X:[0,\infty )\rightarrow \mathbb {H}$, or a trajectory. We also assume that X is smooth enough, e.g., twice continuously differentiable in time. For a fixed $t\in (0,\infty )$, the assumed smoothness on X, together with the identification $t= {\sqrt{s}}k$ and Taylor’s formula in a normed vector space (e.g., [17, Theorem 7.9-1]) implies:

$$\begin{aligned} \frac{x_{k+1}-x_k}{{\sqrt{s}}}&=\dot{X}(t)+\frac{1}{2}\ddot{X}(t){\sqrt{s}}+{\text {o}}\bigl ( {\sqrt{s}} \bigr ) \quad \text { as } s\rightarrow 0, \nonumber \\ \frac{x_{k}-x_{k-1}}{{\sqrt{s}}}&=\dot{X}(t)-\frac{1}{2}\ddot{X}(t){\sqrt{s}}+{\text {o}}\bigl ( {\sqrt{s}} \bigr ) \quad \text { as } s\rightarrow 0, \nonumber \\ {\sqrt{s}}{\mathcal {L}^{-1}}G'(y_k)&={\sqrt{s}}{\mathcal {L}^{-1}}G'(X(t))+{\text {o}}\bigl ( {\sqrt{s}} \bigr ) \quad \text { as } s\rightarrow 0. \end{aligned}$$

(A.1)

The last identity follows from the continuity of $G'$, that of ${\mathcal {L}^{-1}}$, and (3.2), from which we can deduce $y_k\rightarrow X(t)$ as $s\rightarrow 0$. Plugging (3.2) into (3.3) and dividing by ${\sqrt{s}}$, we have $\frac{x_{k+1}-x_k}{{\sqrt{s}}}- \lambda \frac{x_{k}-x_{k-1}}{{\sqrt{s}}}+{\sqrt{s}}{\mathcal {L}^{-1}}G'(y_k)=0$. Substituting the above Taylor expansions, and then rearranging, we arrive at

$$\begin{aligned} \frac{1}{2}(1+\lambda ) \ddot{X}(t) +\frac{1-\lambda }{{\sqrt{s}}}\dot{X}(t)+ {\mathcal {L}^{-1}}G'(X(t))+{\text {o}}\bigl ( 1 \bigr )=0 \quad \text { as } s\rightarrow 0. \end{aligned}$$

(A.2)

To make this estimate consistent, interpret $\lambda$ as a function of s and further assume that $(1-\lambda )/{\sqrt{s}}\rightarrow 2\eta$ as ${\sqrt{s}}\rightarrow 0$ for some $\eta \in (0,\infty )$, which yields

$$\begin{aligned} \ddot{X}(t)+2\eta \dot{X}(t)+{\mathcal {L}^{-1}}G'(X(t))=0. \end{aligned}$$

(A.3)

1.2 A.2. Derivation of the Initial Conditions

The initialization $y_0 = x_0$ and (3.3) with $k=0$ imply

$$\begin{aligned} \frac{x_1-x_0}{{\sqrt{s}}}={\sqrt{s}}{\mathcal {L}^{-1}}G'(x_0) . \end{aligned}$$

Take the limit $s\rightarrow 0$ and conclude $\dot{X}(0)=0$ since $G'$ and $\dot{X}$ are assumed to be continuous. Therefore, we arrive at the desired IVP (4.1).

Remark A.1

(momentum method) A similar procedure can be carried out far more easily for the so-called momentum method (MM). To see this, we recall that

$$\begin{aligned} \ddot{X}(t)\approx \frac{x_{k+1}-2x_k+x_{k-1}}{s}, \quad \dot{X}(t)\approx \frac{x_k-x_{k-1}}{{\sqrt{s}}}, \quad G'(X(t))\approx G'(x_k) . \end{aligned}$$

Then, the discrete version of the ODE (4.1) becomes

$$\begin{aligned} x_{k+1}=x_k-sG'(x_k)+(1-2\eta {\sqrt{s}})(x_k-x_{k-1}), \end{aligned}$$

which is MM with the weight $1-2\eta {\sqrt{s}}$; see [28, p. 12 (9)]. This weight is close to $\lambda$:

$$\begin{aligned} \lambda =\frac{1-\eta {\sqrt{s}}}{1+\eta {\sqrt{s}}}=1-\frac{2\eta {\sqrt{s}}}{1+\eta {\sqrt{s}}}\approx 1-2\eta {\sqrt{s}}. \end{aligned}$$

In this sense, MM seems more natural and amenable for analysis than AGD. $\square$

The limiting behavior of MM can also be explained by the IVP (4.1). Observe that the only essential difference between MM and PAGD is where $G'$ is evaluated, that is, $x_k$ and $y_k$ respectively. And in the limit $s\rightarrow 0$, $x_k$ and $y_k$ are not distinguishable in this setting. However, PAGD exhibits less oscillation than MM since evaluating $G'$ at $y_k$ serves as “foreseeing” the uphill of the objective functional, if exists, along the trajectory and “steering” to avoid unnecessary oscillating behaviors. Recently, a higher order Taylor expansion turns out to help differentiate their performaces (see [31]).

Appendix 2: PAGD as a Discretization of the IVP

Let us label the step size ${\sqrt{s}}$, rather than s, in order to make the setting more in line with the PAGD algorithm. Again, it is helpful to have in mind the correspondence: time $t\longleftrightarrow k{\sqrt{s}}$ ($k=0,1,2,\ldots$) and position $X(t)\longleftrightarrow x_k$. First, we will see $y_k$ corresponds to a “drifted” position without the potential landscape over $[t,t+{\sqrt{s}}]$. This can be modeled by $\ddot{X}(t)+2\eta \dot{X}(t)=0$, which leads to another energy law $\frac{1}{2}\bigl \Vert \dot{X}(t+{\sqrt{s}}) \bigr \Vert _{\mathcal {L}}^2 = \frac{1}{2}\bigl \Vert \dot{X}(t) \bigr \Vert _{\mathcal {L}}^2 -2\eta \int _t^{t+{\sqrt{s}}}\bigl \Vert \dot{X}(\tau ) \bigr \Vert _{\mathcal {L}}^2{d }\tau .$ Approximate the speed in the integrand by the average $\frac{1}{2}(\bigl \Vert \dot{X}(t+{\sqrt{s}}) \bigr \Vert _{\mathcal {L}}+\bigl \Vert \dot{X}(t) \bigr \Vert _{\mathcal {L}})$, then after a short calculation, one obtains $\bigl \Vert \dot{X}(t+{\sqrt{s}}) \bigr \Vert _{\mathcal {L}}=\lambda \bigl \Vert \dot{X}(t) \bigr \Vert _{\mathcal {L}}$. Since the dynamics takes place in a single direction, this implies $\dot{X}(t+{\sqrt{s}})=\lambda \dot{X}(t)$. The approximations $\dot{X}(t)\approx \frac{x_k-x_{k-1}}{{\sqrt{s}}}$ and $\dot{X}(t+{\sqrt{s}})=\frac{y_k-x_k}{{\sqrt{s}}}$ lead us to (3.2).

Next, we discretize the vector V(t). Since we do not know the minimizer in practice, we remove it from the definition of $v_k$ and discretize $V(t)+x^*=X(t)+\frac{1}{\eta }\dot{X}(t)$. The approximations $X(t)\approx y_k$ and $\dot{X}(t)\approx \frac{y_k-x_k}{{\sqrt{s}}}$ suggest

$$\begin{aligned} v_k = y_{k} +\frac{1}{\theta }(y_k-x_{k}), \end{aligned}$$

(B.1)

which leads to the definition of $\{v_k\}_{k\ge 1}$ (3.4) upon combining with the definition of $\{y_k\}$.

Finally, to get the main iterates, $\{x_{k}\}_{k\ge 1}$, we discretize (4.3) using the approximations $\dot{V} (t) \approx \frac{v_{k+1}-v_k}{{\sqrt{s}}}$, $\dot{X} (t) \approx \frac{y_{k}-x_k}{{\sqrt{s}}}$, and the evaluation of $G'$ at $y_k$, then it follows $\eta \frac{v_{k+1}-v_k}{{\sqrt{s}}}+\eta \frac{y_k-x_k}{{\sqrt{s}}}+{\mathcal {L}^{-1}}G'(y_k)=0$. Plugging in (3.4) and (A.1), one obtains (3.3), the definition of $\{x_k\}_{k\ge 1}$.

Appendix 3: Literature Comparison

We summarize our discussion on the existing literature works, and contrast them with our contributions, in Table 2.

Table 2 We summarize our discussion on the existing literature works, and contrast them with our contributions, in this table