Sharp characterization of optimal minibatch size for stochastic finite sum convex optimization

Abstract

The minibatching technique has been extensively adopted to facilitate stochastic first-order methods because of their computational efficiency in parallel computing for large-scale machine learning and data mining. Indeed, increasing the minibatch size decreases the iteration complexity (number of minibatch queries) to converge, resulting in the decrease of the running time by processing a minibatch in parallel. However, this gain is usually saturated for too large minibatch sizes and the total computational complexity (number of access to an example) is deteriorated. Hence, the determination of an appropriate minibatch size which controls the trade-off between the iteration and total computational complexities is important to maximize performance of the method with as few computational resources as possible. In this study, we define the optimal minibatch size as the minimum minibatch size with which there exists a stochastic first-order method that achieves the optimal iteration complexity and we call such a method the optimal minibatch method. Moreover, we show that Katyusha (in: Proceedings of annual ACM SIGACT symposium on theory of computing vol 49, pp 1200–1205, ACM, 2017), DASVRDA (Murata and Suzuki, in: Advances in neural information processing systems vol 30, pp 608–617, 2017), and the proposed method which is a combination of Acc-SVRG (Nitanda, in: Advances in neural information processing systems vol 27, pp 1574–1582, 2014) with APPA (Cotter et al. in: Advances in neural information processing systems vol 27, pp 3059–3067, 2014) are optimal minibatch methods. In experiments, we compare optimal minibatch methods with several competitors on $L_1$-and $L_2$-regularized logistic regression problems and observe that iteration complexities of optimal minibatch methods linearly decrease as minibatch sizes increase up to reasonable minibatch sizes and finally attain the best iteration complexities. This confirms the computational efficiency of optimal minibatch methods suggested by the theory.

Appendices

Convergence Analysis of Acc-SVRG with APPA

We here provide a detailed description concerning convergence analyses for Acc-SVRG with or without APPA. The following convergence theorem is a special case of $n \ge \sqrt{\kappa }$ in [3].

Theorem A

([3]). Consider Algorithm 1. Assume $n\ge \sqrt{\kappa }$ and $g_i$ are L-smooth and $\mu$-strongly convex functions. Let $p\in (0,1)$ satisfy $\frac{2p(2+p)}{1-p} \le \frac{1}{2}$. Set $b\ge \frac{8\sqrt{\kappa }n}{\sqrt{2}p(n-1)+8\sqrt{\kappa }}$, $\eta = \frac{1}{2L}$, $m = \left\lceil \frac{\sqrt{2\kappa }}{1-p}\log \left( \frac{1-p}{p}\right) \right\rceil$. Then, we get

$$\begin{aligned} {\mathbb {E}}[ f({\tilde{x}}_{s+1}) - f_* ] \le \frac{1}{2}( f({\tilde{x}}_{s}) - f_* ) \end{aligned}$$

and we get the following iteration and total complexities by setting $T=O(\log (1/\epsilon ))$.

$$\begin{aligned} I(A(b),f)&= O\left( \left( \frac{n}{b} + \sqrt{\kappa } \right) \log \left( \frac{1}{\epsilon }\right) \right) , \\ T(A(b),f)&= O\left( \left( n + b\sqrt{\kappa } \right) \log \left( \frac{1}{\epsilon }\right) \right) . \end{aligned}$$

When $n\ge \kappa$, the minibatch size of $b=O(n/\sqrt{\kappa })$ satisfies the assumption in this theorem, thus, we can deduce the first claim in the paper concerning Acc-SVRG without APPA:

$$\begin{aligned} I(A(b),f)&= O( \sqrt{\kappa } \log ( 1/\epsilon ) ), \\ T(A(b),f)&= O( n \log ( 1/\epsilon ) ). \end{aligned}$$

The following lemma is key result to derive a total complexity of an optimization method accelerated by APPA. Concretely, Lemma A gives an iteration complexity of Algorithm 2 ignoring a complexity of an inner-solver ${\mathscr {M}}$.

Lemma A

([18]) Suppose that f is $\mu$-strongly convex and $\gamma \ge 3\mu /2$. Consider Algorithm 2. If a generated sequence $\{w_s\}_{s=1}^{T_c +1}$ satisfies the following

$$\begin{aligned} {\mathbb {E}}[ F_s(w_{s+1}) ] -F_s^* \le \frac{1}{4}\left( \frac{\mu }{\mu +2\gamma }\right) ^{\frac{3}{2}}( F_s(w_{s}) - F_s^* ). \end{aligned}$$

(6)

Then, there exists a positive value $C_0$ which only depend on $w_0$ such that

$$\begin{aligned} {\mathbb {E}}[ f(w_{s}) ] - f_* \le \left( 1-\frac{1}{2}\sqrt{ \frac{\mu }{\mu +2\gamma } }\right) ^s C_0. \end{aligned}$$

(7)

From Theorem A and Lemma A, we can derive complexities of Algorithm 1 with APPA in the case of $n < \kappa$. To do so, we specify complexities of Algorithm 1 for solving subproblems with required accuracy (6) by APPA. Let $\gamma = \frac{L}{n} - \mu$. Then, the condition number of subproblems are $\frac{L+\gamma }{\mu +\gamma }= O(n)$. When $\kappa > 3n$, $\gamma > 3\mu /2$ follows. From Theorem A, we find that the required inequality (6) is satisfied with following parameters:

$$\begin{aligned} T= O\left( \log \left( \gamma /\mu \right) \right) ,\ b = O(\sqrt{n}),\ m = O(\sqrt{n}),\ \eta = \frac{1}{2(L+\gamma )}. \end{aligned}$$

Therefore, the iteration and total complexities for each run of Acc-SVRG in APPA are $O(\sqrt{n}\log (\gamma /\mu ))$ and $O(n\log (\gamma /\mu ))$. Since the required number of iterates $T_c$ of APPA to achieve $\epsilon$-accurate solution is $O\left( \sqrt{\frac{\gamma }{\mu }}\log \left( \frac{1}{\epsilon }\right) \right) = O\left( \sqrt{\frac{\kappa }{n}}\log \left( \frac{1}{\epsilon }\right) \right)$. Therefore, we get

$$\begin{aligned} I(A(b),f)&= O\left( \sqrt{\kappa } \log \left( \frac{1}{\epsilon } \right) \log \left( \frac{\kappa }{n}\right) \right) , \\ T(A(b),f)&= O\left( \sqrt{n\kappa } \log \left( \frac{1}{\epsilon } \right) \log \left( \frac{\kappa }{n}\right) \right) . \end{aligned}$$

Experimental details

Here, the details of the experiments in Sect. 6 are provided.

The details of the implemented algorithms and their tuning parameters were as follows:

• SVRG [15]. We tuned the number of inner iterations m and the learning rate $\eta$.

• Acc-SVRG [3]. The number of inner iterations m and the learning rate $\eta$ were tuned. We fixed the momentum parameter $\beta$ as $(1 - \sqrt{\eta \lambda _2})/(1 + \sqrt{\eta \lambda _2})$, where $\lambda _2$ is the $L_2$-regularization parameter².

• SVRG with APPA (SVRG + Algorithm 2). In Addition to the above-described tuning parameters, the positive parameter $\gamma$ described in Algorithm 2 was tuned. We fixed $\mu$ as the $L_2$-regularization parameter $\lambda _2$ $^2$.

• Acc-SVRG with APPA (Acc-SVRG + Algorithm 2). In the same manner as Acc-SVRG and SVRG with APPA, m, $\eta$ and $\gamma$ are tuned.

• DASVRDA [2]. We tuned the number of inner iterations m, the learning rate $\eta$, and additionally the restart interval S only for strongly convex objectives.

For tuning the above parameters, we automatically chose the values from appropriate ranges for each setting, by selecting the ones that led to the minimum objective value after running 50 passes over the data. The used parameter ranges were as follows:

• Learning rate $\eta$: $\{0.1, 0.5, 1.0, 5.0, 10.0\}$.

• The number of inner iterations m:

  $\{n/b, 5n/b, 10n/b, 50n/b, 100n/b\}$, where n is the size of data set and b is the minibatch size.

• APPA parameter $\gamma$: $\{10^k \times \lambda _2 \mid k \in \{0, 1, 2, 3, 4\} \}$, where $\lambda _2$ is the $L_2$-regularization parameter$^2$.

• Restart Interval S: $\{1, 5, 10, 50, 100\}$ (only for $\lambda _2 > 0$).