We develop and analyse a variant of the SARAH algorithm, which does not require computation of the exact gradient. Thus this new method can be applied to general expectation minimization problems rather than only finite sum problems. While the original SARAH algorithm, as well as its predecessor, SVRG, requires an exact gradient computation on each outer iteration, the inexact variant of SARAH (iSARAH), which we develop here, requires only stochastic gradient computed on a mini-batch of sufficient size. The proposed method combines variance reduction via sample size selection and iterative stochastic gradient updates. We analyse the convergence rate of the algorithms for strongly convex and non-strongly convex cases, under smooth assumption with appropriate mini-batch size selected for each case. We show that with an additional, reasonable, assumption iSARAH achieves the best-known complexity among stochastic methods in the case of non-strongly convex stochastic functions.

我们开发并分析了SARAH算法的一种变体，它不需要计算精确的梯度。因此，这种新方法可以应用于一般期望最小化问题，而不仅仅是有限和问题。虽然原始的SARAH算法及其前身SVRG需要在每次外部迭代中进行精确的梯度计算，但我们在此处开发的SARAH（iSARAH）的不精确变体只需要在一个足够小批量下计算出的随机梯度尺寸。所提出的方法结合了通过样本大小选择和迭代随机梯度更新的方差减少。在平滑假设下，针对每种情况选择适当的最小批处理大小，我们分析了强凸和非强凸情况下算法的收敛速度。我们通过额外的合理证明