SIAM Journal on Optimization, Volume 30, Issue 1, Page 960-979, January 2020.
We study constrained nested stochastic optimization problems in which the objective function is a composition of two smooth functions whose exact values and derivatives are not available. We propose a single timescale stochastic approximation algorithm, which we call the nested averaged stochastic approximation (NASA), to find an approximate stationary point of the problem. The algorithm has two auxiliary averaged sequences (filters) which estimate the gradient of the composite objective function and the inner function value. By using a special Lyapunov function, we show that the NASA achieves the sample complexity of ${\cal O}(1/\epsilon^{2})$ for finding an $\epsilon$-approximate stationary point, thus outperforming all extant methods for nested stochastic approximation. Our method and its analysis are the same for both unconstrained and constrained problems, without any need of batch samples for constrained nonconvex stochastic optimization. We also present a simplified parameter-free variant of the NASA method for solving constrained single-level stochastic optimization problems, and we prove the same complexity result for both unconstrained and constrained problems.

中文翻译：

---

嵌套随机优化的单一时间尺度随机逼近方法

SIAM优化杂志，第30卷，第1期，第960-979页，2020年1月。
我们研究约束嵌套的随机优化问题，其中目标函数是两个光滑函数的组合，而两个光滑函数的精确值和导数不可用。我们提出了一种单一的时间尺度随机逼近算法，我们将其称为嵌套平均随机逼近（NASA），以找到问题的近似平稳点。该算法有两个辅助平均序列（滤波器），用于估计复合目标函数和内部函数值的梯度。通过使用特殊的Lyapunov函数，我们发现NASA可以找到$ {\ cal O}（1 / \ epsilon ^ {2}）$的样本复杂度，以找到近似于$ \ epsilon $的固定点，从而胜过所有现存的嵌套随机逼近的方法。对于无约束和受约束的问题，我们的方法及其分析都是相同的，不需要任何批处理样本即可进行受约束的非凸随机优化。我们还提出了一种简化的无参数NASA方法变体，用于解决约束单级随机优化问题，并且对于无约束和受约束的问题，我们证明了相同的复杂性结果。