Analysis of biased stochastic gradient descent using sequential semidefinite programs

Abstract

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix inequality (LMI) whose feasible points lead to convergence bounds of biased SGD. Based on this LMI condition, we develop a sequential minimization approach to analyze the intricate trade-offs that couple stepsize selection, convergence rate, optimization accuracy, and robustness to gradient inaccuracy. We also provide feasible points for this LMI and obtain theoretical formulas that quantify the convergence properties of biased SGD under various assumptions on the loss functions.

Appendices

Appendix

Proof of Theorem 1

First notice that since $i_k$ is uniformly distributed on $\{1,\dots ,n\}$ and $x_k$ and $i_k$ are independent, we have:

$$\begin{aligned} \,{\mathbb {E}} \bigl ( u_k \,\big |\, x_k \bigr ) = \,{\mathbb {E}} \bigl ( \nabla f_{i_k}(x_k) \,\big |\, x_k \bigr ) = \frac{1}{n}\sum _{i=1}^n \nabla f_i(x_k) = \nabla g(x_k) \end{aligned}$$

Consequently, we have:

$$\begin{aligned} \,{\mathbb {E}} \left( \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}^{\mathsf {T}}\! \begin{bmatrix} -2mI_p &{} I_p \\ I_p &{} 0_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}\bigg |\ x_k\right) = \begin{bmatrix} x_k-x_\star \\ \nabla g(x_k) \end{bmatrix}^{\mathsf {T}}\! \begin{bmatrix} -2m I_p &{} I_p\\ I_p &{} 0_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ \nabla g(x_k) \end{bmatrix} \ge 0 \end{aligned}$$

(54)

where the inequality in (54) follows from the definition of $g \in {\mathcal {S}}(m,\infty )$.

Next we prove (12), let’s start with Case I, the boundedness constraint $\Vert \nabla f_i(x_k)\Vert \le \beta$ implies that $\left\| u_k\right\| \le \beta$ for all k. Rewrite as a quadratic form to obtain:

$$\begin{aligned} \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}^{\mathsf {T}}\begin{bmatrix} 0_p &{}\quad 0_p \\ 0_p &{}\quad -I_p \end{bmatrix} \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix} \ge -\beta ^2 \end{aligned}$$

(55)

The boundedness constraint $\Vert \nabla f_i(x_k)-mx_k\Vert \le \beta$ implies that:

$$\begin{aligned} \left\| u_k-m(x_k-x_\star )\right\| ^2&\le \left\| (u_k-mx_k)+mx_\star \right\| ^2 + \left\| (u_k-mx_k)-mx_\star \right\| ^2\\&=2\left\| u_k-mx_k\right\| ^2 + 2m^2\left\| x_\star \right\| ^2 \\&\le 2\beta ^2 + 2m^2\left\| x_\star \right\| ^2 \end{aligned}$$

As in the proof of Case I, rewrite the above inequality as a quadratic form and we obtain the second row of Table 1.

To prove the three remaining cases, we begin by showing that an inequality of the following form holds for each $f_i$:

$$\begin{aligned} \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)-\nabla f_i(x_\star ) \end{bmatrix}^{\mathsf {T}}\begin{bmatrix} M_{11}I_p &{}\quad M_{12}I_p \\ M_{21}I_p &{}\quad -2I_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)-\nabla f_i(x_\star ) \end{bmatrix} \ge 0 \end{aligned}$$

(56)

The verification for (56) follows directly from the definitions of L-smoothness and convexity. In the smooth case (Definition 1), for example, $\Vert \nabla f_i(x_k)-\nabla f_i(x_\star )\Vert \le L\Vert x_k-x_\star \Vert$. So (56) holds with $M_{11}=2L^2$, $M_{12}=M_{21}=0$. The cases for ${\mathcal {F}}(0,L)$ and ${\mathcal {F}}(m,L)$ follow directly from Definition 2. In Table 1, we always have $M_{22}=-1$. Therefore,

$$\begin{aligned}&\,{\mathbb {E}} \left( \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}^{\mathsf {T}}\begin{bmatrix} M_{11}I_p &{} M_{12}I_p \\ M_{21}I_p &{} M_{22}I_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}\,\,\bigg |\ \, x_k\right) \nonumber \\&\quad =\frac{1}{n} \sum _{i=1}^n \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)\end{bmatrix}^{\mathsf {T}}\begin{bmatrix} M_{11}I_p &{} M_{12}I_p \\ M_{21}I_p &{} 0_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)\end{bmatrix}- \frac{1}{n} \sum _{i=1}^n \Vert \nabla f_i(x_k)\Vert ^2 \end{aligned}$$

(57)

Since $\frac{1}{n}\sum _{i=1}^n \nabla f_i(x_\star ) = \nabla g(x_\star ) = 0$, the first term on the right side of (57) is equal to

$$\begin{aligned} \frac{1}{n} \sum _{i=1}^n \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)-\nabla f_i(x_\star )\end{bmatrix} \begin{bmatrix} M_{11}I_p &{} M_{12}I_p \\ M_{21}I_p &{} 0_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ \nabla f_i(x_k)-\nabla f_i(x_\star ) \end{bmatrix} \end{aligned}$$

Based on the constraint condition (56), we know that the above term is greater than or equal to $\frac{2}{n} \sum _{i=1}^n \Vert \nabla f_i(x_k)-\nabla f_i(x_\star ) \Vert ^2$. Substituting this fact back into (57) leads to the inequality:

$$\begin{aligned}&\,{\mathbb {E}} \left( \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}^{\mathsf {T}}\begin{bmatrix} M_{11}I_p &{} M_{12}I_p \\ M_{21}I_p &{} M_{22}I_p\end{bmatrix} \begin{bmatrix} x_k-x_\star \\ u_k \end{bmatrix}\,\,\bigg |\ \, x_k\right) \nonumber \\&\quad \ge \frac{1}{n} \sum _{i=1}^n \left( 2\Vert \nabla f_i(x_k)-\nabla f_i(x_\star ) \Vert ^2- \Vert \nabla f_i(x_k)\Vert ^2\right) \nonumber \\&\quad = \frac{1}{n} \sum _{i=1}^n \left( \Vert \nabla f_i(x_k)-2\nabla f_i(x_\star ) \Vert ^2- 2\Vert \nabla f_i(x_\star )\Vert ^2\right) \nonumber \\&\quad \ge -\frac{2}{n} \sum _{i=1}^n \Vert \nabla f_i(x_\star )\Vert ^2 \end{aligned}$$

(58)

Taking the expectation of both sides, we arrive at (12), as desired. Now we are ready to prove our main theorem. By Schur complement, (10) is equivalent to (15), which can be further rewritten as

$$\begin{aligned} \left( \begin{bmatrix} 1-\rho _k^2 &{}\quad -\alpha _k &{}\quad -\alpha _k\\ alpha_k &{}\quad \alpha _k^2 &{}\quad \alpha _k^2 \\ alpha_k &{}\quad \alpha _k^2 &{}\quad \alpha _k^2 \end{bmatrix} +\nu _k\!\begin{bmatrix} -2m &{}\quad 1 &{}\quad 0\\ 1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 \end{bmatrix} +\lambda _k\!\begin{bmatrix} M_{11} &{}\quad M_{12} &{}\quad 0\\ M_{21} &{}\quad M_{22} &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 \end{bmatrix} +\mu _k\!\begin{bmatrix} 0 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad \delta ^2 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad -1 \end{bmatrix}\right) \otimes I_p\preceq 0 \end{aligned}$$

(59)

Since $x_{k+1}-x_\star =x_k-x_\star -\alpha _k(u_k+e_k)$, we have

$$\begin{aligned} \begin{bmatrix} x_k-x_\star \\ u_k \\ e_k \end{bmatrix}^{\mathsf {T}}\left( \begin{bmatrix} 1 &{}\quad -\alpha _k &{}\quad -\alpha _k\\ alpha_k &{}\quad \alpha _k^2 &{}\quad \alpha _k^2 \\ -\alpha _k &{}\quad \alpha _k^2 &{}\quad \alpha _k^2 \end{bmatrix}\otimes I_p\right) \begin{bmatrix} x_k-x_\star \\ u_k \\ e_k \end{bmatrix}=\Vert x_{k+1}-x_\star \Vert ^2 \end{aligned}$$

(60)

Now we can left and right multiply (59) by $[(x_k-x_\star )^{\mathsf {T}}, u_k^{\mathsf {T}}, e_k^{\mathsf {T}}]$ and $[(x_k-x_\star )^{\mathsf {T}}, u_k^{\mathsf {T}}, e_k^{\mathsf {T}}]^{\mathsf {T}}$, and apply the inequalities (4), (54), and (12) to get the desired conclusion. $\square$

Proof of Lemma 2

We use an induction argument to prove Item 1. For simplicity, we denote (48) as $V_{k+1}=h(V_k)$. Suppose we have $V_k=h(V_{k-1})$ and $V_{k-1}>V_\star$. We are going to show $V_{k+1}=h(V_k)$ and $V_k>V_\star$. We can rewrite (48) as

$$\begin{aligned} V_{k+1}=\,\, \mathop {{{\,\mathrm{minimize}\,}}}\limits _{\zeta > 0} \quad A_k(1+Z_k^{-1}) + B_k(1+Z_k) \end{aligned}$$

(61)

where $A_k$, $B_k$, and $Z_k$ are defined as

$$\begin{aligned} A_k&=\frac{m^2 V_k^2 \left( c^2+(2G^2+{\tilde{M}} V_k)\delta ^2\right) }{(2G^2+{\tilde{M}} V_k)^2}\\ B_k&=\frac{(2G^2 V_k+({\tilde{M}}-m^2) V_k^2)((2G^2+{\tilde{M}} V_k)(1-\delta ^2)-c^2)}{(2G^2+{\tilde{M}} V_k)^2}\\ Z_k&=\frac{\bigl (2 G^2+{\tilde{M}} V_k\bigr )(\delta ^2+\zeta _k )+c^2}{(2G^2+{\tilde{M}} V_k)(1-\delta ^2)-c^2} \end{aligned}$$

Note that $A_k \ge 0$ and $B_k \ge 0$ due to the condition $2G^2(1-\delta ^2) \ge c^2$. The objective in (61) therefore has a form very similar to the objective in (26). Applying Lemma 1, we deduce that $V_{k+1} =(\sqrt{A_k}+\sqrt{B_k})^2$, which is the same as (49). The associated $Z_k^opt$ is $\sqrt{\tfrac{A_k}{B_k}}$. To ensure this is a feasible choice, it remains to check that the associated $\zeta _k^opt > 0$ as well. Via algebraic manipulations, one can show that $\zeta _k > 0$ is equivalent to $V_k > V_\star$. We can also verify $A_k$ is a monotonically increasing function of $V_k$, and $B_k$ is a monotonically nondecreasing function of $V_k$. Hence h is a monotonically increasing function. Also notice $V_\star$ is a fixed point of (49). Therefore, if we assume $V_k=h(V_{k-1})$ and $V_{k-1}>V_\star$, we have $V_k=h(V_{k-1})>h(V_\star )=V_\star$. Hence we guarantee $\zeta _k>0$ and $V_{k+1}=h(V_k)$. By similar arguments, one can verify $V_1=h(V_0)$. And it is assumed that $V_0>V_\star$. This completes the induction argument.

Item 2 follows from a similar argument to the one used in Sect. 2.2. Finally, Item 3 can be proven by choosing a sufficiently small constant stepsize $\alpha$ to make ${\hat{U}}_k$ arbitrarily close to $V_\star$. Since $V_\star \le V_k \le {\hat{U}}_k$, we conclude that $\lim _{k\rightarrow \infty } V_k = V_\star$, as required. $\square$