Variance reduction techniques provide simple and fast algorithms for solving machine learning problems. In this paper, we present a novel stochastic variance-reduced method. The proposed method relies on the mini-batch version of stochastic recursive gradient algorithm (MB-SARAH), which updates stochastic gradient estimates by using a simple recursive scheme. However, facing the challenge of the step size sequence selection in MB-SARAH, we introduce an online step size sequence based on the hypergradient descent (HD) method, which only requires little additional computation. For the proposed method, referred to as MB-SARAH-HD, we provide a general convergence analysis and prove linear convergence for strongly convex problems in expectation. Specifically, we prove that the proposed method has sublinear convergence rate in a single outer loop. We also prove that the iteration complexity outperforms several variants of the state-of-the-art stochastic gradient descent (SGD) method under suitable conditions. Numerical experiments on standard datasets are provided to demonstrate the efficacy and superiority of our MB-SARAH-HD method over existing approaches in the literature.

方差减少技术提供了用于解决机器学习问题的简单快速算法。在本文中，我们提出了一种新颖的减少随机方差的方法。所提出的方法基于随机递归梯度算法的最小批量版本（MB-SARAH），该算法通过使用简单的递归方案来更新随机梯度估计。但是，面对MB-SARAH中步长序列选择的挑战，我们引入了基于超梯度下降（HD）方法的在线步长序列，该方法仅需要很少的额外计算。对于所提出的称为MB-SARAH-HD的方法，我们提供了一般的收敛性分析，并证明了期望中的强凸问题的线性收敛性。具体来说，我们证明了该方法在单个外部环路中具有亚线性收敛速度。我们还证明，在合适的条件下，迭代复杂度优于最新的随机梯度下降（SGD）方法的几种变体。提供了标准数据集上的数值实验，以证明我们的MB-SARAH-HD方法相对于现有文献中方法的有效性和优越性。