A note on solving nonlinear optimization problems in variable precision

Abstract

This short note considers an efficient variant of the trust-region algorithm with dynamic accuracy proposed by Carter (SIAM J Sci Stat Comput 14(2):368–388, 1993) and by Conn et al. (Trust-region methods. MPS-SIAM series on optimization, SIAM, Philadelphia, 2000) as a tool for very high-performance computing, an area where it is critical to allow multi-precision computations for keeping the energy dissipation under control. Numerical experiments are presented indicating that the use of the considered method can bring substantial savings in objective function’s and gradient’s evaluation “energy costs” by efficiently exploiting multi-precision computations.

Appendix: Complexity theory for the TR1DA algorithm

For the sake of accuracy and completeness, we now provide details of the first-order worst-case complexity analyis summarized at the end of Sect. 2. As indicated there, the following development can be seen as a combination of the arguments proposed by [16] for the convergence theory of trust-region methods with inexact gradients (pp. 280) and dynamic accuracy (pp. 400).

We assume that

AS.1::

The objective function f is twice continuously differentiable in $\mathfrak {R}^n$ and there exist a constant $\kappa _\nabla \ge 0$ such that $\Vert \nabla _x^2f(x)\Vert \le \kappa _\nabla$ for all $x \in \mathfrak {R}^n$.

AS.2::

There exists a constant $\kappa _H\ge 0$ such that $\Vert H_k\Vert \le \kappa _H$ for all $k\ge 0$.

AS.3:

There exists a constant $\kappa _{\mathrm{low}}$ such that $f(x)\ge \kappa _{\mathrm{low}}$ for all $x\in \mathfrak {R}^n$.

Lemma A.1

Suppose AS.1 and AS.2 hold. Then, for each $k\ge 0,$

$$\begin{aligned} |f(x_k+s_k)-m(x_k,s_k)| \le |f_k-f(x_k)| +\kappa _g \Vert \overline{g}(x_k,\omega _{g,k})\Vert \Delta _k + \kappa _{H\nabla }\Delta _k^2. \end{aligned}$$

(A.1)

for $\kappa _{H\nabla } = 1+\max [\kappa _H, \kappa _\nabla ].$

Proof

(See [16, Theorem 8.4.2].) The definition (2.8), (2.6), the mean-value theorem, the Cauchy–Schwarz inequality and AS.1 give that, for some $\xi _k$ in the segment $[x_k,x_k+s_k]$,

$$\begin{aligned} |f(x_k+s_k)-m(x_k,s_k)| &\le |f_k-f(x_k)| +|s_k^T(\nabla _x^1f(x_k)-\overline{g}(x_k,\omega _{g,k})| \\&\quad+ { \frac{1}{2}}|s_k^T\nabla _x^2f(\xi _k)s_k| + { \frac{1}{2}}|s_k^TH_ks_k| \\ &\le |f_k-f(x_k)| +\kappa _g \Vert \overline{g}(x_k,\omega _{g,k})\Vert \, \Vert s_k\Vert + { \frac{1}{2}}(\kappa _H + \kappa _\nabla )\Vert s_k\Vert ^2 \end{aligned}$$

and (A.1) follows from the the Cauchy–Schwarz inequality and the inequality $\Vert s_k\Vert \le \Delta _k$. $\square$

Lemma A.2

We have that, for all $k\ge 0,$

$$\begin{aligned} \max \left[ |f_k-f(x_k)|,|f_k^+ - f(x_k+s_k)|\right] \le \eta _0\left[ m(x_k,0)-m(x_k,s_k)\right] \end{aligned}$$

(A.2)

and

$$\begin{aligned} \rho _k \ge \eta _1 \;\; \text{ implies } \text{ that } \;\; \frac{f(x_k)-f(x_k+s_k)}{m(x_k,0)-m(x_k,s_k)} \ge \eta _1 - 2\eta _0 >0. \end{aligned}$$

(A.3)

Proof

(See [16, p. 401].) The mechanism of the TR1DA algorithm ensures that (A.2) holds. Hence,

$$\begin{aligned} \frac{\left[ f_k-f(x_k)\right] +\left[ |f_k^+ -f(x_k+s_k)\right] }{m(x_k,0)-m(x_k,s_k)} \le 2 \eta _0. \end{aligned}$$

As a consequence, for iterations where $\rho _k \ge \eta _1$,

$$\begin{aligned} \rho _k = \frac{f_k-f_k^+}{m(x_k,0)-m(x_k,s_k)} = \frac{f(x_k)-f(x_k+s_k)}{m(x_k,0)-m(x_k,s_k)} + \frac{\left[ f_k-f(x_k)\right] +\left[ |f_k^+-f(x_k+s_k)\right] }{m(x_k,0)-m(x_k,s_k)} \end{aligned}$$

and (A.3) must hold. $\square$

This result implies, in particular, that the sequence $\{f(x_k)\}_{k\ge 0}$ is non-increasing, and the TR1DA algorithm is therefore monotone on the exact function f.

Lemma A.3

Suppose AS.1 and AS.2 hold, and that $\overline{g}(x_k,\omega _{g,k})\ne 0.$ Then

$$\begin{aligned} \Delta _k \le \frac{\Vert \overline{g}(x_k,\omega _{g,k})\Vert }{2\kappa _{H\nabla }} \Big [ { \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g\Big ] \;\; \text{ implies } \text{ that } \;\; \Delta _{k+1} \ge \Delta _k. \end{aligned}$$

(A.4)

Proof

(See [16, Theorem 8.4.3].) Since (2.5) implies that ${ \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g \in (0,1)$ the first part of (A.4) then gives that $\Delta _k < \Vert \overline{g}(x_k,\omega _{g,k})\Vert\, /\, \kappa _{H\nabla }$. Hence the inequality $1+\Vert H_k\Vert \le \kappa _{H\nabla }$ and (2.9) yield that

$$\begin{aligned} m(x_k,0)-m(x_k,s_k) \ge { \frac{1}{2}}\Vert \overline{g}(x_k,\omega _{g,k})\Vert \min \left[ \frac{\Vert \overline{g}(x_k,\omega _{g,k})\Vert }{\kappa _{H\nabla }},\Delta _k\right] = { \frac{1}{2}}\Vert \overline{g}(x_k,\omega _{g,k})\Vert \,\Delta _k. \end{aligned}$$

As a consequence, we may use (2.11), the Cauchy–Schwarz inequality, (A.2) (twice), (A.1), the inequality $\kappa _{H\nabla }\ge 1$ and the first part of (A.4) to deduce that, for all $k\ge 0$,

$$\begin{aligned} \begin{array}{lcl} |\rho _k-1| &{} = &{} \frac{ |f_k^+- m(x_k,s_k)|}{ m(x_k,0)-m(x_k,s_k)}\\ &{} \le &{} \frac{ |f_k^+-f(x_k+s_k)|+|f(x_k+s_k)-m(x_k,s_k)|}{ m(x_k,0)-m(x_k,s_k)}\\ &{} \le &{} 2 \eta _0 +\frac{ \kappa _g\Vert \overline{g}(x_k,\omega _{g,k})\Vert \Delta _k+\kappa _{H\nabla }\Delta _k^2}{ { \frac{1}{2}}\Vert \overline{g}(x_k,\omega _{g,k})\Vert \,\Delta _k}\\ &{} \le &{} 2\eta _0 +2\kappa _g+ 2\kappa _{H\nabla }\frac{ \Delta _k}{ \Vert \overline{g}(x_k,\omega _{g,k})\Vert }\\ &{} \le &{} 1-\eta _2. \end{array} \end{aligned}$$

Thus $\rho _k\ge \eta _2$ and (2.12) ensures the second part of (A.4). $\square$

Lemma A.4

Suppose that AS.1 and AS.2 hold. Then, before termination,

$$\begin{aligned} \Delta _k \ge \min \left[ \Delta _0,\theta \epsilon \right] \;\; \text{ where } \;\; 0< \theta {\mathop {=}\limits ^{\mathrm{def}}}\frac{\gamma _1\big [{ \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g\big ]}{\kappa _{H\nabla }(1+\kappa _g)} <\frac{1}{\kappa _{H\nabla }(1+\kappa _g)}. \end{aligned}$$

(A.5)

Proof

(See [16, Theorem 8.4.4].) Before termination, we must have that

$$\begin{aligned} \Vert \overline{g}(x_k,\omega _{g,k})\Vert \ge \frac{\epsilon }{1+\kappa _g}. \end{aligned}$$

(A.6)

Suppose that iteration k is the first iteration such that

$$\begin{aligned} \Delta _{k+1} \le \frac{\gamma _1\epsilon }{\kappa _{H\nabla }(1+\kappa _g)} \big [{ \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g\big ]. \end{aligned}$$

(A.7)

Then the update (2.12) implies that

$$\begin{aligned} \Delta _k \le \frac{\epsilon }{\kappa _{H\nabla }(1+\kappa _g)} \big [{ \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g\big ] \le \frac{\Vert \overline{g}(x_k,\omega _{g,k})\Vert }{\kappa _{H\nabla }} \big [{ \frac{1}{2}}(1-\eta _1)-\eta _0-\kappa _g\big ] \end{aligned}$$

where we have used (A.6) to deduce the last inequality. But this bound and (A.4) imply that $\Delta _{k+1}\ge \Delta _k$, which is impossible since $\Delta _k$ is reduced at iteration k. Hence no k exists such that (A.7) holds and the desired conclusion follows. $\square$

Lemma A.5

For each $k \ge 0,$ define

$$\begin{aligned} \mathcal{S}_k {\mathop {=}\limits ^{\mathrm{def}}}\{ j \in \{ 0, \ldots , k \} \mid \rho _j \ge \eta _1 \} \;\; \text{ and } \;\; \mathcal{U}_k {\mathop {=}\limits ^{\mathrm{def}}}\{ 0, \ldots , k \} \setminus \mathcal{S}_k. \end{aligned}$$

(A.8)

the index sets of “successful” and “unsuccessful” iterations, respectively. Then

$$\begin{aligned} k \le |\mathcal{S}_k| \left( 1-\frac{\log \gamma _3}{\log \gamma _2}\right) +\frac{1}{|\log \gamma _2|}\log \left( \frac{\Delta _0}{\theta \epsilon }\right) . \end{aligned}$$

(A.9)

Proof

Observe that (2.12) implies that

$$\begin{aligned} \Delta _{j+1} \le \gamma _3 \Delta _j \;\; \text{ for } \;\; j \in \mathcal{S}_k \end{aligned}$$

and that

$$\begin{aligned} \Delta _{j+1}\le \gamma _2\Delta _j \;\; \text{ for } \;\; j \in \mathcal{U}_k. \end{aligned}$$

Combining these two inequalities, we obtain from (A.5) that

$$\begin{aligned} \min \big [\Delta _0,\theta \epsilon \Big ]\le \Delta _k \le \Delta _0\gamma _3^{|S_k|}\gamma _2^{|U_k|} \end{aligned}$$

Dividing by $\Delta _0$, taking logarithms and recalling that $\gamma _2\in (0,1)$, we get

$$\begin{aligned} |\mathcal{U}_k| \le -|\mathcal{S}_k| \frac{\log \gamma _3}{\log \gamma _2} - \frac{1}{\log \gamma _2}\log \left( \frac{\Delta _0}{\theta \epsilon }\right) . \end{aligned}$$

Hence (A.9) follows since $k = |\mathcal{S}_k|+|\mathcal{U}_k|$. $\square$

Theorem A.1

Suppose that AS.1–AS.3 hold. Suppose also that$\Delta _0\ge \theta \epsilon,$where$\theta$is defined in (A.5). Then the TR1DA algorithm produces an iterate$x_k$such that$\Vert \nabla _x^1f(x_k)\Vert \le \epsilon$ in at most

$$\tau _{{\mathcal{S}}} \mathop = \limits^{{{\text{def}}}} \frac{{2(f(x_{0} ) - \kappa _{{{\text{low}}}} )(1 + \kappa _{g} )}}{{(\eta _{1} - 2\eta _{0} )\theta }} \cdot \frac{1}{{\epsilon^{2} }}$$

(A.10)

successful iterations, at most

$$\begin{aligned} \tau _{\mathrm{tot}} {\mathop {=}\limits ^{\mathrm{def}}}\tau _S \left( 1-\frac{\log \gamma _3}{\log \gamma _2}\right) +\frac{1}{|\log \gamma _2|}\log \left( \frac{\Delta _0}{\theta \epsilon }\right) \end{aligned}$$

(A.11)

iterations in total, at most$\tau _{\mathrm{tot}}$(approximate) evaluations of the gradient satisfying (2.6), and at most$2\tau _{\mathrm{tot}}$(approximate) evaluations of the objective function satisfying (2.2).

Proof

As in the previous proof, (A.6) must hold before termination. Using AS.3, (A.8), (A.3), (2.9), (A.6), the assumption that $\Delta _0\ge \theta \epsilon$ and (A.5), we obtain that, for an arbitrary $k\ge 0$ before termination,

$$\begin{aligned} f(x_0)-\kappa _{\mathrm{low}}& \ge \sum _{j\in \mathcal{S}_k}\left[ f(x_j)-f(x_{j+1})\right] \\& \ge (\eta _1-2\eta _0) \sum _{j\in \mathcal{S}_k}\left[ m(x_j,0)-m(x_j,s_j)\right] \\& \ge { \frac{1}{2}}(\eta _1-2\eta _0) \sum _{j\in \mathcal{S}_k}\Vert \overline{g}(x_j,\omega _{g,j})\Vert \min \left[ \frac{ \Vert \overline{g}(x_j,\omega _{g,j})\Vert }{ 1+\Vert H_j\Vert },\Delta _j\right] \\& \ge { \frac{1}{2}}|\mathcal{S}_k|(\eta _1-2\eta _0)\frac{ \epsilon }{ 1+\kappa _g} \min \left[ \frac{ \epsilon }{ \kappa _{H\nabla }(1+\kappa _g)},\min \Big [\Delta _0,\theta \epsilon \Big ]\right] \\& = |\mathcal{S}_k| \,\frac{ (\eta _1-2\eta _0)}{ 2(1+\kappa _g)} \min \left[ \frac{ 1}{ \kappa _{H\nabla }(1+\kappa _g)},\theta \right] \,\epsilon ^2 \\& = |\mathcal{S}_k|\,\frac{ (\eta _1-2\eta _0)\theta }{ 2(1+\kappa _g)} \, \epsilon ^2 \end{aligned}$$

and therefore

$$|{\mathcal{S}}_{k} | \le \frac{{2(f(x_{0} ) - \kappa _{{{\text{low}}}} )(1 + \kappa _{g} )}}{{(\eta _{1} - 2\eta _{0} )\theta }} \cdot \frac{1}{{\epsilon ^{2} }}\mathop = \limits^{{{\text{def}}}} \frac{{\tau _{{\mathcal{S}}} }}{{\epsilon ^{2} }}.$$

As a consequence $\Vert \overline{g}(x_k,\omega _{g,k})\Vert < \epsilon /(1+\kappa _g)$ after at most $\tau _S\epsilon ^{-2}$ successful iterations and the algorithm terminates. The relation (2.13) then ensures that $\Vert \nabla _x^1f(x_k)\Vert < \epsilon$, yielding (A.10). We may now use (A.9) and the mechanism of the algorithm to complete the proof. $\square$

Given that $\epsilon \in (0,1]$, we immediately note that

$$\tau _{{\mathcal{S}}} = {\mathcal{O}}(\epsilon ^{{ - 2}} )\;\;{\text{ and }}\;\;\tau _{{{\text{tot}}}} = {\mathcal{O}}(\epsilon ^{{ - 2}} ).$$

Moreover, the proof of Theorem A.1 implies that these complexity bounds improve from $\mathcal{O}(\epsilon ^{-2})$ to $\mathcal{O}(\epsilon ^{-1})$ if $\epsilon$ is so large or $\Delta _0$ so small to yield $\Delta _0 < \theta \epsilon$.

We conclude this brief complexity theory by noting that the domain in which AS.1 is assumed to hold can be restricted to the “tree of iterates” $\cup _{k\ge 0}[x_k,x_k+s_k]$ without altering our results. This can be useful if an upper bound ${\bar{\Delta }}$ is imposed on the step’s length, in which case the monotonicty of the algorithm ensures that the tree of iterates remains in the set

$$\begin{aligned} \{y \in \mathfrak {R}^n \,|\, y = x+s \;\; \text{ with } \;\; f(x)\le f(x_0) \;\; \text{ and } \;\; \Vert s\Vert \le {\bar{\Delta }} \}. \end{aligned}$$

While it can be difficult to verify AS.1 on the (a priori unpredictable) tree of iterates, verifying it on the above set is much easier.