Proximal Methods Avoid Active Strict Saddles of Weakly Convex Functions

Abstract

We introduce a geometrically transparent strict saddle property for nonsmooth functions. This property guarantees that simple proximal algorithms on weakly convex problems converge only to local minimizers, when randomly initialized. We argue that the strict saddle property may be a realistic assumption in applications, since it provably holds for generic semi-algebraic optimization problems.

Appendices

Proofs of Theorems 2.9 and 5.2

In this section, we prove Theorem 2.9. We should note that Theorem 2.9, appropriately restated, holds much more broadly beyond the weakly convex function class. To simplify the notational overhead, however, we impose the weak convexity assumption, throughout.

We will require some basic notation from variational analysis; for details, we refer the reader to [57]. A set-valued map $F:\mathbb {R}^d\rightrightarrows \mathbb {R}^m$ assigns to each point $x\in \mathbb {R}^d$ a set F(x) in $\mathbb {R}^m$. The graph of F is defined by

$$\begin{aligned} \mathrm{gph}\,F:=\{(x,v):v\in F(x)\}.\end{aligned}$$

A map $F:\mathbb {R}^d\rightrightarrows \mathbb {R}^m$ is called metrically regular at $(\bar{x},\bar{v})\in \mathrm{gph}\,F$ if there exists a constant $\kappa >0$ such that the estimate holds:

$$\begin{aligned} \mathrm{dist}(x,F^{-1}(v))\le \kappa \mathrm{dist}(v,F(x))\end{aligned}$$

for all x near $\bar{x}$ and all v near $\bar{v}$. If the graph $\mathrm{gph}\,F$ is a $C^1$-smooth manifold around $(\bar{x},\bar{v})$, then metric regularity at $(\bar{x},\bar{v})$ is equivalent to the condition [57, Theorem 9.43(d)]:¹²

$$\begin{aligned} (0,u)\in N_{\mathrm{gph}\,F}(\bar{x},\bar{v})\quad \Longrightarrow \quad u=0. \end{aligned}$$

(A.1)

We begin with the following lemma.

Lemma A.1

(Subdifferential metric regularity in smooth minimization). Consider the optimization problem

$$\begin{aligned} \min _{x\in \mathbb {R}^d} f(x)\quad \text {subject to}\quad x\in \mathcal {M},\end{aligned}$$

where $f:\mathbb {R}^d\rightarrow \mathbb {R}$ is a $C^2$-smooth function and $\mathcal {M}$ is a $C^2$-smooth manifold. Let $\bar{x}\in \mathcal {M}$ satisfy the criticality condition $0\in \partial f_{\mathcal {M}}(\bar{x})$ and suppose that the subdifferential map $\partial f_{\mathcal {M}}:\mathbb {R}^d\rightrightarrows \mathbb {R}^d$ is metrically regular at $(\bar{x},0)$. Then, the guarantee holds:

$$\begin{aligned} \inf _{u\in \mathbb {S}^{d-1}\cap T_{\mathcal {M}}(\bar{x})} d^2 f_{\mathcal {M}}(\bar{x})(u)\ne 0. \end{aligned}$$

(A.2)

Proof

First, appealing to (A.1), we conclude that the implication holds:

$$\begin{aligned} (0,u)\in N_{\mathrm{gph}\,\partial f_{\mathcal {M}}}(\bar{x},0)\quad \Longrightarrow \quad u=0. \end{aligned}$$

(A.3)

Let us now interpret the condition (A.3) in Lagrangian terms. To this end, let $G=0$ be the local defining equations for $\mathcal {M}$ around $\bar{x}$. Define the Lagrangian function

$$\begin{aligned} \mathcal {L}(x,\lambda )=f(x)+\langle G(x),\lambda \rangle ,\end{aligned}$$

and let $\bar{\lambda }$ be the unique Lagrange multiplier vector satisfying $\nabla _x \mathcal {L}(\bar{x},\bar{\lambda })=0$. According to [41, Corollary 2.9], we have the following expression:

$$\begin{aligned} (0,u)\in N_{\mathrm{gph}\,\partial f_{\mathcal {M}}}(\bar{x},0)\quad \Longleftrightarrow \quad u\in T_{\mathcal {M}}(\bar{x})\quad \text {and}\quad L u \in N_{\mathcal {M}}(\bar{x}), \end{aligned}$$

(A.4)

where $L:=\nabla ^2_{xx}\mathcal {L}(\bar{x},\bar{\lambda })$ denotes the Hessian of the Lagrangian. Combining (A.3) and (A.4), we deduce that the only vector $u\in T_{\mathcal {M}}(\bar{x})$ satisfying $L u\in N_{\mathcal {M}}(\bar{x})$ is the zero vector $u=0$.

Now for the sake of contradiction, suppose that (A.2) fails. Then, the quadratic form $Q(u)=\langle L u,u\rangle$ is nonnegative on $T_{\mathcal {M}}(\bar{x})$ and there exists $0\ne \bar{u}\in T_{\mathcal {M}}(\bar{x})$ satisfying $Q(\bar{u})=0$. We deduce that $\bar{u}$ minimizes $Q(\cdot )$ on $T_{\mathcal {M}}(\bar{x})$, and therefore, the inclusion $L\bar{u}\in N_{\mathcal {M}}(\bar{x})$ holds, a clear contradiction. $\square$

The following corollary for active manifolds will now quickly follow.

Corollary A.2

(Subdifferential metric regularity and active manifolds). Consider a closed and weakly convex function $f:\mathbb {R}^d\rightarrow \mathbb {R}\cup \{\infty \}$. Suppose that f admits a $C^2$-smooth active manifold around a critical point $\bar{x}$ and that the subdifferential map $\partial f:\mathbb {R}^d\rightrightarrows \mathbb {R}^d$ is metrically regular at $(\bar{x},0)$. Then, $\bar{x}$ is either a strong local minimizer of f or satisfies the curvature condition $d^2 f_{\mathcal {M}}(\bar{x})(u)<0$ for some $u\in T_{\mathcal {M}}(\bar{x})$.

Proof

The result [19, Proposition 10.2] implies that $\mathrm{gph}\,\partial f$ coincides with $\mathrm{gph}\,\partial f_{\mathcal {M}}$ on a neighborhood of $(\bar{x},0)$. Therefore, the subdifferential map $\partial f_{\mathcal {M}}:\mathbb {R}^d\rightrightarrows \mathbb {R}^d$ is metrically regular at $(\bar{x},0)$. Using Lemma A.1, we obtain the guarantee:

$$\begin{aligned} \inf _{u\in \mathbb {S}^{d-1}\cap T_{\mathcal {M}}(\bar{x})} d^2 f_{\mathcal {M}}(\bar{x})(u)\ne 0. \end{aligned}$$

If the infimum is strictly negative, the proof is complete. Otherwise, the infimum is strictly positive. In this case, $\bar{x}$ is a strong local minimizer of $f_{\mathcal {M}}$, and therefore by [19, Proposition 7.2] a strong local minimizer of f. $\square$

We are now ready for the proofs of Theorems 2.9 and 5.2.

Proof of Theorem 2.9

The result [18, Corollary 4.8] shows that for almost all $v\in \mathbb {R}^d$, the function $g(x):=f(x)-\langle v,x\rangle$ has at most finitely many critical points. Moreover each such critical point $\bar{x}$ lies on some $C^2$ active manifold $\mathcal {M}$ of g and the subdifferential map $\partial g:\mathbb {R}^d\rightrightarrows \mathbb {R}^d$ is metrically regular at $(\bar{x},0)$. Applying Corollary A.2 to g for such generic vectors v, we deduce that every critical point $\bar{x}$ of g is either a strong local minimizer or a strict saddle of g. The proof is complete. $\square$

Proof of Theorem 5.2

The proof is identical to that of Theorem 2.9 with [18, Theorem 5.2] playing the role of [18, Corollary 4.8]. $\square$

Pathological Example

Theorem B.1

Consider the following function

$$\begin{aligned} f(x, y) = \frac{1}{2}(|x| + |y|)^2 - \frac{\rho }{2} x^2 \end{aligned}$$

Assume that $\lambda > \rho$. Define a mapping $T :\mathbb {R}^d \rightarrow \mathbb {R}$ by the following formula.

$$\begin{aligned} S(x, y) = {\left\{ \begin{array}{ll} 0 &{} \text {if }(x, y) = 0;\\ \left( 0, \frac{\lambda }{1+\lambda } y\right) &{} \text {if }|x| \le \frac{1}{1+\lambda } |y|; \\ \left( \frac{\lambda }{1+\lambda - \rho }x, 0\right) &{} \text {if }|y| \le \frac{1}{1+\lambda -\rho }|x|,\\ \end{array}\right. } \end{aligned}$$

and if $\frac{1}{ (1+\lambda - \rho )} |x|< |y| < (1+\lambda ) |x|$, we have

$$\begin{aligned} S(x,y) = {\left\{ \begin{array}{ll} \frac{\lambda }{(1+\lambda )(1+\lambda - \rho )- 1} \begin{bmatrix} (1+\lambda ) &{} -1 \\ -1 &{} (1+\lambda - \rho ) \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} &{} \text {if }\mathrm {sign}(x) = \mathrm {sign}(y);\\ \frac{\lambda }{(1+\lambda )(1+\lambda - \rho )-1} \begin{bmatrix} (1+\lambda ) &{} 1 \\ 1 &{} (1+\lambda - \rho ) \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix}&\text {if }\mathrm {sign}(x) \ne \mathrm {sign}(y). \end{array}\right. } \end{aligned}$$

Then, $\mathrm{prox}_{(1/\lambda ) f}(x, y) = S(x,y)$.

Proof

Let us denote the components of S(x, y) by $(x_+, y_+) = S(x,y)$. By first-order optimality conditions, we have $\mathrm{prox}_{(1/\lambda ) f}(x, y) = (x_+, y_+)$ if and only if

$$\begin{aligned}&\lambda (x - (1-(1/\lambda )\rho )x_+, y - y_+) \in \\&{\left\{ \begin{array}{ll} \{x_+ + \mathrm {sign}(x_+)|y_+|\}\times \{\mathrm {sign}(y_+)|x_+| + y_+\} &{} \text {if }x_+ \ne 0\text { and }y_+ \ne 0;\\ ([-1, 1]y_+)\times \{y_+\} &{} \text {if }x_+ = 0\text { and }y_+ \ne 0;\\ \{x_+\}\times ([-1, 1]x_+) &{} \text {if }x_+ \ne 0\text { and }y_+ = 0;\\ \{0\}\times \{0\} &{} \text {if }x_+ = 0\text { and }y_+ = 0.\\ \end{array}\right. } \end{aligned}$$

Let us show that $(x_+, y_+)$ indeed satisfies this inclusion.

• 1. If $(x,y) = 0$, then $x_+ = y_+ = 0$, and the pair satisfies the inclusion.

• 2. If $|x| \le \frac{1}{1 + \lambda }|y|$ and $y \ne 0$, then $x_+ = 0$, $y_+ = \frac{\lambda }{1+\lambda }y$, and

  $$\begin{aligned} \lambda (x - (1-(1/\lambda )\rho )x_+, y - y_+) = \lambda \left( x, \frac{1}{1 + \lambda }y\right) \in ([-1, 1]y_+) \times \{y_+\}. \end{aligned}$$

  Thus, the pair satisfies the inclusion.

• 3. If $|y| \le \frac{1}{1+\lambda -\rho }|x|$ and $x \ne 0$, then $x_+ = \frac{\lambda }{(1+\lambda - \rho )}x$, $y_+ = 0$, and

  $$\begin{aligned}&\lambda (x - (1-(1/\lambda )\rho )x_+, y - y_+)\\&= \lambda \left( x - \frac{\lambda -\rho }{(1+\lambda - \rho )}x, y\right) \in \{x_+\}\times ([-1, 1]x_+). \end{aligned}$$

For the remaining two cases, let us assume that $\frac{1}{ (1+\lambda - \rho )} |x|< |y| < (1+\lambda ) |x|$.

• 4. If $\mathrm {sign}(x) = \mathrm {sign}(y)$, let $s = \mathrm {sign}(x)$ and note that

  $$\begin{aligned} \begin{bmatrix} x_+ \\ y_+ \end{bmatrix}&= \frac{\lambda }{(1+\lambda )(1+\lambda - \rho )- 1} \begin{bmatrix} (1+\lambda ) &{} -1 \\ -1 &{} (1+\lambda - \rho ) \end{bmatrix}\begin{bmatrix} x\\ y \end{bmatrix}\\&= \frac{s\lambda }{(1+\lambda )(1+\lambda - \rho )- 1} \begin{bmatrix} (1+\lambda )|x| -|y| \\ -|x| + (1+\lambda - \rho )|y| \end{bmatrix} \end{aligned}$$

  From this equation we learn $\mathrm {sign}(x_+) = \mathrm {sign}(y_+) = s$. Inverting the matrix, we also learn

  $$\begin{aligned} \lambda \begin{bmatrix} x \\ y \end{bmatrix} =\begin{bmatrix} (1+\lambda - \rho ) &{} 1 \\ 1 &{} (1+\lambda ) \end{bmatrix} \begin{bmatrix} x_+ \\ y_+ \end{bmatrix}&= \begin{bmatrix} x_+ + \lambda (1 - \rho /\lambda )x_+ + y_+ \\ x_+ + y_+ + \lambda y_+ \end{bmatrix} \\&= \begin{bmatrix} x_+ + \mathrm {sign}(x_+) |y_+| + \lambda (1 - \rho /\lambda )x_+ \\ \mathrm {sign}(y_+) |x_+| + y_+ + \lambda y_+ \end{bmatrix}. \end{aligned}$$

  Thus, the pair satisfies the inclusion.

• 5. If $\mathrm {sign}(x) \ne \mathrm {sign}(y)$, let $s = \mathrm {sign}(x)$ and note that

  $$\begin{aligned} \begin{bmatrix} x_+ \\ y_+ \end{bmatrix}&= \frac{\lambda }{(1+\lambda )(1+\lambda - \rho )- 1} \begin{bmatrix} (1+\lambda ) &{} 1 \\ 1 &{} (1+\lambda - \rho ) \end{bmatrix}\begin{bmatrix} x\\ y \end{bmatrix}\\&= \frac{s\lambda }{(1+\lambda )(1+\lambda - \rho )- 1} \begin{bmatrix} (1+\lambda )|x| -|y| \\ |x| - (1+\lambda - \rho )|y| \end{bmatrix} \end{aligned}$$

  From this equation we learn $\mathrm {sign}(x_+) \ne \mathrm {sign}(y_+)$. Inverting the matrix we also learn

  $$\begin{aligned} \lambda \begin{bmatrix} x \\ y \end{bmatrix} =\begin{bmatrix} (1+\lambda - \rho ) &{} -1 \\ -1 &{} (1+\lambda ) \end{bmatrix} \begin{bmatrix} x_+ \\ y_+ \end{bmatrix}&= \begin{bmatrix} x_+ + \lambda (1 - \rho /\lambda )x_+ - y_+ \\ -x_+ + y_+ + \lambda y_+ \end{bmatrix} \\&= \begin{bmatrix} x_+ + \mathrm {sign}(x_+) |y_+| + \lambda (1 - \rho /\lambda )x_+ \\ \mathrm {sign}(y_+) |x_+| + y_+ + \lambda y_+ \end{bmatrix}. \end{aligned}$$

  Thus, the pair satisfies the inclusion.

Therefore, the proof is complete. $\square$

Corollary B.2

(Convergence to Saddles). Assume the setting of Theorem B.1. Let $\alpha \in (0, 1]$ and define the operator $T : = (1-\alpha ) I + \alpha S$ on $\mathbb {R}^2$. Then, the cone $\mathcal {K}= \{(x,y) :|x| \le (1+\lambda )^{-1}y\}$ satisfies $T\mathcal {K}\subseteq \mathcal {K}$. Moreover, for any $(x, y) \in \mathcal {K}$, it holds that $T^k(x,y) = ((1-\alpha )^k x, (1 - \alpha (1 - \lambda (1+\lambda )^{-1}))^ky)$ linearly converges to the origin as k tends to infinity.

Proof

Since $\mathcal {K}$ is convex, it suffices to show that $S \mathcal {K}\subseteq \mathcal {K}$. This follows from Theorem B.1. $\square$