Incremental Without Replacement Sampling in Nonconvex Optimization

Abstract

Minibatch decomposition methods for empirical risk minimization are commonly analyzed in a stochastic approximation setting, also known as sampling with replacement. On the other hand, modern implementations of such techniques are incremental: they rely on sampling without replacement, for which available analysis is much scarcer. We provide convergence guaranties for the latter variant by analyzing a versatile incremental gradient scheme. For this scheme, we consider constant, decreasing or adaptive step sizes. In the smooth setting, we obtain explicit complexity estimates in terms of epoch counter. In the nonsmooth setting, we prove that the sequence is attracted by solutions of optimality conditions of the problem.

Appendices

Appendix

This is the appendix for “Incremental Without Replacement Sampling in Nonconvex Optimization.” We begin with the proof of the first claim of the paper.

Proof of Claim 1

We have for all $K \in \mathbb {N}$ and $i = 1 \ldots n$, using the recursion in Algorithm 1,

$$\begin{aligned} z_{K,i} - x_K = \sum _{j=1}^i \alpha _{K,j} d\left( \hat{z}_{K,j-1} \right) . \end{aligned}$$

Using Lemma A.1, we obtain

$$\begin{aligned} \Vert z_{K,i} - x_K\Vert ^2 \le i \sum _{j=1}^i \alpha _{K,i}^2 \Vert d\left( \hat{z}_{K,i-1} \right) \Vert ^2 \le n \sum _{i=1}^n \alpha _{K,i}^2 \Vert d\left( \hat{z}_{K,i-1} \right) \Vert ^2. \end{aligned}$$

Taking $i = n$, we obtain the second inequality. The result follows for $\hat{z}_{K, i-1}$ because it is in $\mathrm {conv}(z_{K,j})_{j=0}^{i-1}$ and

$$\begin{aligned}&\Vert \hat{z}_{K,i-1} - x_K\Vert ^2 \le \max _{z \in \mathrm {conv}(z_{K,j})_{j=0}^{i-1}} \Vert z - x_K\Vert ^2 \\&\quad = \max _{j = 0, \ldots ,i} \Vert z_{K,j} - x_K\Vert ^2 \le n \sum _{i=1}^n \alpha _{K,i}^2 \Vert d\left( \hat{z}_{K,i-1} \right) \Vert ^2, \end{aligned}$$

where the equality in the middle follows because the maximum of a convex function over a polyhedra is achieved at vertices. $\square$

A Proofs for the Smooth Setting

For all $K \in \mathbb {N}$, we let $\alpha _K = \alpha _{K-1,n}$, with $\alpha _0 = \delta ^{-1/3} \ge \alpha _{0,1}$.

1.1 A.1 Analysis for Both Step Size Strategies

Claim A.1

We have for all $K \in \mathbb {N}$,

$$\begin{aligned}&\left\langle \nabla F(x_K), x_{K+1} - x_{K} \right\rangle + \frac{1}{2n \alpha _K} \Vert x_{K+1} - x_K\Vert ^2\nonumber \\&\quad \le - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \alpha _K L^2n^2 \sum _{j=1}^n \alpha _{K,j}^2\Vert d_j(\hat{z}_{K,j-1})\Vert ^2 \nonumber \\&\qquad + \alpha _K M^2 \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2. \end{aligned}$$

(19)

Proof of Claim A.1

Fix $K \in \mathbb {N}$, we have

$$\begin{aligned} x_{K+1} - x_K = - \sum _{i=1}^n \alpha _{K,i} d_i(\hat{z}_{K,i-1}) = - \alpha _K \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1}) \end{aligned}$$

(20)

Recall that $\nabla F(x_K) = \frac{1}{n} \sum _{i=1}^n d_i(x_K)$, combining with (20), we deduce the following

$$\begin{aligned}&\left\langle \nabla F(x_K), x_{K+1} - x_{K} \right\rangle + \frac{1}{2n \alpha _K} \Vert x_{K+1} - x_K\Vert ^2 \nonumber \\&\quad =\frac{-\alpha _K}{n} \left\langle \sum _{i=1}^n d_i(x_K), \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1}) \right\rangle + \frac{1}{2n \alpha _K} \Vert x_{K+1} - x_K\Vert ^2\nonumber \\&\quad = \frac{\alpha _K}{2n} \left( \left\| \sum _{i=1}^n d_i(x_K) - \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1}) \right\| ^2 - \left\| \sum _{i=1}^n d_i(x_K)\right\| ^2 - \left\| \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1}) \right\| ^2\right) \nonumber \\&\qquad + \frac{1}{2n \alpha _K} \Vert x_{K+1} - x_K\Vert ^2\nonumber \\&\quad = - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \frac{\alpha _K}{2n} \left\| \sum _{i=1}^n d_i(x_K) - \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1}) \right\| ^2 \nonumber \\&\quad \le - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \frac{\alpha _K}{n} \left( \left\| \sum _{i=1}^n d_i(x_K) - \sum _{i=1}^n d_i(\hat{z}_{K,i-1})\right\| ^2\right. \nonumber \\&\qquad + \left. \left\| \sum _{i=1}^n d_i(\hat{z}_{K,i-1}) - \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1})\right\| ^2\right) , \end{aligned}$$

(21)

where the first two equalities are properties of the scalar product, the third equality uses (20) to drop canceling terms and the last inequality uses $\Vert a + b \Vert ^2 \le 2(\Vert a\Vert ^2 + \Vert b\Vert ^2)$. We bound each term separately, first,

$$\begin{aligned} \left\| \sum _{i=1}^n d_i(x_K) - \sum _{i=1}^n d_i(\hat{z}_{K,i-1}) \right\| ^2&\le \left( \sum _{i=1}^n \Vert d_i(x_K) -d_i(\hat{z}_{K,i-1}) \Vert \right) ^2 \nonumber \\&\le \left( \sum _{i=1}^n L_i\Vert x_K -\hat{z}_{K,i-1}\Vert \right) ^2 \nonumber \\&\le \max _{i=1,\ldots , n}\Vert x_K -\hat{z}_{K,i-1}\Vert ^2 \left( \sum _{i=1}^n L_i \right) ^2 \nonumber \\&\le L^2n^3 \sum _{j=1}^n \alpha _{K,j}^2 \Vert d_j(\hat{z}_{K,j-1})\Vert ^2. \end{aligned}$$

(22)

where the first step uses the triangle inequality, the second step uses $L_i$ Lipschicity of $d_i$, the third step is Hölder inequality, and the fourth step uses Claim 1. Furthermore, we have using the triangle inequality and Cauchy–Schwartz inequality,

$$\begin{aligned} \left\| \sum _{i=1}^n d_i(\hat{z}_{K,i-1}) - \sum _{i=1}^n \frac{\alpha _{K,i}}{\alpha _{K}} d_i(\hat{z}_{K,i-1})\right\| ^2&\le \left( \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) \Vert d_i(\hat{z}_{K,i-1})\Vert \right) ^2 \nonumber \\&\le \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2 \sum _{i=1}^n \Vert d_i(\hat{z}_{K,i-1})\Vert ^2\nonumber \\&\le \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2 \sum _{i=1}^n M_i^2 \nonumber \\&= nM^2 \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2. \end{aligned}$$

(23)

Combining (21), (22) and (23), we obtain,

$$\begin{aligned}&\left\langle \nabla F(x_K), x_{K+1} - x_{K} \right\rangle + \frac{1}{2n \alpha _K} \Vert x_{K+1} - x_K\Vert ^2\nonumber \\&\quad \le - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \alpha _K L^2n^2 \sum _{j=1}^n \alpha _{K,j}^2\Vert d_j (\hat{z}_{K,j-1})\Vert ^2 + \alpha _K M^2 \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2, \end{aligned}$$

which is (19) $\square$

Claim A.2

F has L Lipschitz gradient.

Proof

For any x, y, we have

$$\begin{aligned} \Vert \nabla F(x) - \nabla F(y)\Vert&= \frac{1}{n} \left\| \sum _{i=1}^n d_i(x) - d_i(y) \right\| \le \frac{1}{n} \sum _{i=1}^n \left\| d_i(x) - d_i(y) \right\| \le \frac{1}{n} \sum _{i=1}^n L_i\left\| x - y \right\| \\&= L \Vert x - y \Vert \end{aligned}$$

where we used triangle inequality and $L_i$ Lipschicity of $d_i$. $\square$

Proof of Claim 2

Using smoothness of F in Claim A.2, we have from the descent Lemma [43, Lemma 1.2.3], for all $x,y \in \mathbb {R}^p$

$$\begin{aligned} F(y) \le F(x) + \left\langle \nabla F(x), y - x \right\rangle + \frac{L}{2} \Vert y-x \Vert ^2. \end{aligned}$$

(24)

Choosing $y = x_{K+1}$ and $x = x_K$ in (24), using Claims A.1 and 1, we obtain

$$\begin{aligned} F(x_{K+1})&\le F(x_K) + \left\langle \nabla F(x_K), x_{K+1} - x_K \right\rangle + \frac{L}{2} \Vert x_{K+1} - x_K\Vert ^2\\&\le F(x_K) - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \alpha _K L^2n^2\sum _{j=1}^n \alpha _{K,j}^2\Vert d_j(\hat{z}_{K,j-1})\Vert ^2 \\&\quad + \alpha _K M^2 \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2 + \left( \frac{L}{2} - \frac{1}{2n\alpha _K}\right) \Vert x_{K+1} - x_K\Vert ^2 \\&\le F(x_K) - \frac{n \alpha _K}{2} \Vert \nabla F(x_K) \Vert ^2 + \left( \alpha _K L^2n^2+ \frac{Ln}{2} - \frac{1}{2 \alpha _K}\right) \sum _{j=1}^n \alpha _{K,j}^2\Vert d_j(\hat{z}_{K,j-1})\Vert ^2 \\&\quad + \alpha _K M^2 \sum _{i=1}^n \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2. \end{aligned}$$

Since $\alpha _{K,i} \le \alpha _K$ for all $K \in \mathbb {N}$ and $i =1 \ldots , n$, we have $0 \le \alpha _{K,i} / \alpha _K \le 1$, and using $(t-1)^2 \le 1 - t^2$ for all $t \in [0,1]$

$$\begin{aligned} \left( \frac{\alpha _{K,i}}{\alpha _{K}} - 1 \right) ^2&\le 1 - \frac{\alpha _{K,i}^2}{\alpha _K^2}\le 1 - \frac{\alpha _{K,i}^3}{\alpha _K^3}, \end{aligned}$$

and the result follows. $\square$

B Proofs for the Nonsmooth Setting

Proof of Theorem 1

Fix $T > 0$, we consider the sequence of functions, for each $k \in \mathbb {N}$

$$\begin{aligned} \mathbf {w}_k :[0,T]&\mapsto \mathbb {R}^p \\ t&\mapsto \mathbf {w}(\tau _k + t). \end{aligned}$$

From Assumption 4 and Definition 5, it is clear that all functions in the sequence are M Lipschitz. Since the sequence $(x_k)_{k \in \mathbb {N}}$ is bounded, $(\mathbf {w}_k)_{k\in \mathbb {N}}$ is also uniformly bounded, hence by Arzelà–Ascoli theorem [51, Chapter 10, Lemma 2], there is a subsequence converging uniformly, let $\mathbf {z}:[0,T] \mapsto \mathbb {R}^p$ be any such uniform limit. By discarding terms, we actually have $\mathbf {w}_k \rightarrow \mathbf {z}$ as $k \rightarrow \infty$, uniformly on [0, T]. Note that we have for all $t \in [0,1]$, and all $\gamma >0$

$$\begin{aligned} D^{\gamma }(\mathbf {w}_k(t)) \subset D^{\gamma + \Vert \mathbf {w}_k - \mathbf {z}\Vert _\infty }(\mathbf {z}(t)). \end{aligned}$$

(25)

For all $k \in \mathbb {N}$, we set $\mathbf {v}_k \in L^2([0,T], \mathbb {R}^p)$ such that $\mathbf {v}_k = \mathbf {w}_k'$ at points where $\mathbf {w}_k$ is differentiable (almost everywhere since it is piecewise affine). We have for all $k \in \mathbb {N}$ and all $s \in [0,T]$

$$\begin{aligned} \mathbf {w}_k(s) - \mathbf {w}_k(0) = \int _{t=0}^{t=s} \mathbf {v}_k(t)\mathrm{d}t, \end{aligned}$$

(26)

and from Definition 5, we have for almost all $t \in [0,T]$,

$$\begin{aligned} \mathbf {v}_k(t) \in - D^{\gamma (\tau _k + t)}(\mathbf {w}_k(t)). \end{aligned}$$

(27)

Hence, the functions $\mathbf {v}_k$ are uniformly bounded thanks to Assumption 4 and hence the sequence $(\mathbf {v}_k)_{k\in \mathbb {N}}$ is bounded in $L^2([0,T], \mathbb {R}^p)$ and by Banach–Alaoglu theorem [51, Section 15.1], it has a weak cluster point. Denote by $\mathbf {v}$ a weak limit of $\left( \mathbf {v}_k \right) _{k \in \mathbb {N}}$ in $L^2([0,T], \mathbb {R}^p)$. Discarding terms, we may assume that $\mathbf {v}_k \rightarrow \mathbf {v}$ weakly in $L^2([0,T], \mathbb {R}^p)$ as $k \rightarrow \infty$ and hence, passing to the limit in (26), for all $s \in [0,T]$,

$$\begin{aligned} \mathbf {z}(s) - \mathbf {z}(0) = \int _{t=0}^{t=s} \mathbf {v}(t)\mathrm{d}t. \end{aligned}$$

(28)

By Mazur’s Lemma (see for example [26]), there exists a sequence $(N_k)_{k \in \mathbb {N}}$, with $N_k \ge k$ and a sequence $\tilde{\mathbf {v}}_{k \in \mathbb {N}}$ such that for each $k \in \mathbb {N}$, $\tilde{\mathbf {v}}_k \in \mathrm {conv}\left( \mathbf {v}_k,\ldots , \mathbf {v}_{N_k} \right)$ such that $\tilde{\mathbf {v}}_k$ converges strongly in $L^2([0,T], \mathbb {R}^p)$ hence pointwise almost everywhere in [0, T]. Using (27) and the fact that countable intersection of full measure sets has full measure, we have for almost all $t \in [0,T]$

$$\begin{aligned} \mathbf {v}(t) = \lim _{k \rightarrow \infty } \tilde{\mathbf {v}}_k(t)&\in \lim _{k \rightarrow \infty } -\mathrm {conv}\left( \cup _{j=k}^{N_k} D^{\gamma (\tau _j + t)}(\mathbf {w}_j(t)) \right) \\&\subset \lim _{k \rightarrow \infty } -\mathrm {conv}\left( \cup _{j=k}^{N_k} D^{\gamma (\tau _j + t) + \Vert \mathbf {w}_j - \mathbf {z}\Vert _\infty }(\mathbf {z}(t)) \right) \\&= - \mathrm {conv}\left( \frac{1}{n} \sum _{i=1}^n D_i(\mathbf {z}(t)) \right) = -D(\mathbf {z}(t)). \end{aligned}$$

where we have used (25), the fact that $\lim _{\gamma \rightarrow 0} D^\gamma = \frac{1}{n} \sum _{i=1}^n D_i$ pointwise since each $D_i$ has closed graph and the definition of D. Using (28), this shows that for almost all $t \in [0,T]$,

$$\begin{aligned} \dot{\mathbf {z}}( t ) = \mathbf {v}(t) \in - D(\mathbf {z}(t)). \end{aligned}$$

Using [7, Theorem 4.1], this shows that $\mathbf {w}$ is an asymptotic pseudotrajectory. $\square$

C Lemmas and Additional Proofs

Lemma A.1

Let $a_1,\ldots , a_m$ be vectors in $\mathbb {R}^p$, then

$$\begin{aligned} \left\| \sum _{i=1}^m a_i \right\| ^2 \le m \sum _{i=1}^m \Vert a_i\Vert ^2. \end{aligned}$$

Proof

From the triangle inequality, we have

$$\begin{aligned} \left\| \sum _{i=1}^m a_i \right\| ^2 \le \left( \sum _{i=1}^m \Vert a_i\Vert \right) ^2. \end{aligned}$$

Hence, it suffices to prove the claim for $p=1$. Consider the quadratic form on $\mathbb {R}^m$

$$\begin{aligned} Q :x \mapsto m \sum _{i=1}^m x_i^2 - \left\| \sum _{i=1}^m x_i \right\| ^2. \end{aligned}$$

We have

$$\begin{aligned} Q(x) = m (\Vert x\Vert ^2 - \left( x^T e \right) ^2), \end{aligned}$$

where $e \in \mathbb {R}^m$ has unit norm and with all entries equal to $1/\sqrt{m}$. The corresponding matrix is $m(I - e e^T)$ which is positive semidefinite. This proves the result. $\square$

Lemma A.2

Let $(a_k)_{k \in \mathbb {N}}$ be a sequence of positive numbers, and $b,c>0$. Then, for all $m \in \mathbb {N}$

$$\begin{aligned} \sum _{i=0}^m \frac{a_i}{b + c \sum _{i=0}^k a_i} \le \frac{1}{c}\log \left( 1 + c \frac{\sum _{i=0}^m a_k}{b} \right) . \end{aligned}$$

Proof

We have

$$\begin{aligned} \sum _{i=0}^m \frac{a_i}{b + c \sum _{i=0}^k a_i}&= \frac{1}{ c} \sum _{i=0}^m \frac{a_i}{\frac{b}{c} + \sum _{i=0}^k a_i} \\&\le \frac{1}{ c} \log \left( 1 + c \frac{\sum _{i=0}^m a_i}{b} \right) \end{aligned}$$

where the last inequality follows from Lemma 6.2 in [24]. $\square$