SIAM Journal on Optimization, Volume 31, Issue 1, Page 244-274, January 2021.
Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and nonconvex, we establish the convergence in the long run of the iterates to a stationary point under a stability condition. The key ingredient is the introduction of a continuous-time version of Adam, under the form of a nonautonomous ordinary differential equation. This continuous-time system is a relevant approximation of the Adam iterates, in the sense that the interpolated Adam process converges weakly toward the solution to the ODE. The existence and the uniqueness of the solution are established. We further show the convergence of the solution toward the critical points of the objective function and quantify its convergence rate under a Łojasiewicz assumption. Then, we introduce a novel decreasing stepsize version of Adam. Under mild assumptions, it is shown that the iterates are almost surely bounded and converge almost surely to critical points of the objective function. Finally, we analyze the fluctuations of the algorithm by means of a conditional central limit theorem.

中文翻译：

---

非凸随机优化的ADAM算法的收敛性和动力学行为

SIAM优化杂志，第31卷，第1期，第244-274页，2021年1月。
亚当是随机梯度下降的流行变体，用于发现函数的局部极小值。在恒定步长体制下，假设目标函数是可微且非凸的，则在稳定条件下，从长远来看，我们建立了收敛到固定点的收敛。关键因素是采用非自治常微分方程的形式引入亚当的连续时间形式。从内插的亚当过程朝ODE的解弱收敛的意义上讲，该连续时间系统是亚当迭代的一个相关近似。确定了解决方案的存在性和唯一性。我们进一步显示了朝向目标函数临界点的解的收敛性，并在Łojasiewicz假设下量化了其收敛速度。然后，我们介绍了一种新颖的递减渐进式亚当版本。在温和的假设下，证明了迭代几乎可以确定地有界，并且几乎可以确定地收敛到目标函数的临界点。最后，我们通过条件中心极限定理分析了算法的波动性。