Large-scale data science trains models for data sets containing massive numbers of samples. Training is often formulated as the solution of empirical risk minimization problems that are optimization programs whose complexity scales with the number of elements in the data set. Stochastic optimization methods overcome this challenge, but they come with their own set of limitations. This article discusses recent developments to accelerate the convergence of stochastic optimization through the exploitation of second-order information. This is achieved with stochastic variants of quasi-Newton methods that approximate the curvature of the objective function using stochastic gradient information. The reasons for why this leads to faster convergence are discussed along with the introduction of an incremental method that exploits memory to achieve a superlinear convergence rate. This is the best-known convergence rate for a stochastic optimization method. Stochastic quasi-Newton methods are applied to several problems, including prediction of the click-through rate of an advertisement displayed in response to a specific search engine query by a specific visitor. Experimental evaluations showcase reductions in overall computation time relative to stochastic gradient descent algorithms.

中文翻译：

---

随机拟牛顿法

大规模数据科学为包含大量样本的数据集训练模型。训练通常被表述为经验风险最小化问题的解决方案，这些问题是优化程序，其复杂性与数据集中元素的数量成比例。随机优化方法克服了这一挑战，但它们也有其自身的局限性。本文讨论了通过利用二阶信息来加速随机优化收敛的最新进展。这是通过使用随机梯度信息近似目标函数曲率的拟牛顿方法的随机变体来实现的。讨论了为什么这会导致更快收敛的原因，同时介绍了一种利用内存来实现超线性收敛速度的增量方法。这是随机优化方法最著名的收敛速度。随机拟牛顿方法应用于若干问题，包括预测响应特定访问者的特定搜索引擎查询而显示的广告的点击率。实验评估展示了相对于随机梯度下降算法的整体计算时间的减少。包括预测响应特定访问者的特定搜索引擎查询而显示的广告的点击率。实验评估展示了相对于随机梯度下降算法的整体计算时间的减少。包括预测响应特定访问者的特定搜索引擎查询而显示的广告的点击率。实验评估展示了相对于随机梯度下降算法的整体计算时间的减少。