The need for statistical estimation with large data sets has reinvigorated interest in iterative procedures and stochastic optimization. Stochastic approximations are at the forefront of this recent development as they yield procedures that are simple, general and fast. However, standard stochastic approximations are often numerically unstable. Deterministic optimization, in contrast, increasingly uses proximal updates to achieve numerical stability in a principled manner. A theoretical gap has thus emerged. While standard stochastic approximations are subsumed by the framework Robbins and Monro (The annals of mathematical statistics, 1951, pp. 400–407), there is no such framework for stochastic approximations with proximal updates. In this paper, we conceptualize a proximal version of the classical Robbins–Monro procedure. Our theoretical analysis demonstrates that the proposed procedure has important stability benefits over the classical Robbins–Monro procedure, while it retains the best known convergence rates. Exact implementations of the proximal Robbins–Monro procedure are challenging, but we show that approximate implementations lead to procedures that are easy to implement, and still dominate standard procedures by achieving numerical stability, practically without trade‐offs. Moreover, approximate proximal Robbins–Monro procedures can be applied even when the objective cannot be calculated analytically, and so they generalize stochastic proximal procedures currently in use.

中文翻译：

---

近端罗宾斯-蒙罗法

对具有大数据集的统计估计的需求重新激发了人们对迭代过程和随机优化的兴趣。随机近似法在最近的发展中处于最前沿，因为它们产生的过程简单，通用和快速。但是，标准随机近似值通常在数值上不稳定。相反，确定性优化越来越多地使用近端更新以原则上的方式实现数值稳定性。因此出现了理论上的差距。标准随机近似被框架Robbins和Monro（数学统计年鉴，1951年，第400-407页），没有采用近端更新的随机近似框架。在本文中，我们概念化了经典Robbins-Monro程序的近端版本。我们的理论分析表明，与传统的Robbins-Monro过程相比，所提出的过程具有重要的稳定性优势，同时保留了最著名的收敛速度。近端Robbins–Monro程序的确切实现具有挑战性，但是我们证明，近似实现会导致易于实现的过程，并且仍然通过实现数值稳定性而在标准过程中占据主导地位，几乎没有取舍。此外，即使无法通过分析计算出目标，也可以应用近似的近端Robbins-Monro程序，