Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai–Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods.

随机方差减少梯度（SVRG）方法是使机器学习和许多其他应用中经常出现的大量成本函数的平均值最小化的重要方法。在本文中，基于SVRG，我们提出了一种SVRG-TR方法，该方法采用类似信任区域的方案来选择步长。证明了SVRG-TR方法对光滑的强凸函数具有线性收敛性，并且比SVRG方法具有更快的收敛速度。为了克服手动调整逐步大小的困难，我们建议结合Barzilai-Borwein（BB）方法来自动计算SVRG-TR方法（称为SVRG-TR-BB方法）的逐步大小。通过将小批量方案分别与SVRG-TR和SVRG-TR-BB结合，我们进一步提出了两种扩展方法mSVRG-TR和mSVRG-TR-BB。给出了mSVRG-TR的线性收敛性和复杂性。在一些标准数据集上的数值实验表明，SVRG-TR和SVRG-TR-BB通常在最佳调整步长和一些现代随机梯度方法方面优于或与SVRG相当，而mSVRG-TR和mSVRG-TR-BB则非常有竞争力最近成功的随机梯度方法的小批量变种。