In this paper, we consider supervised learning problems over training sets in which the number of training examples and the dimension of feature vectors are both large. We focus on the case where the loss function defining the quality of the parameter we wish to estimate may be non-convex, but also has a convex regularization. We propose a Doubly Stochastic Successive Convex approximation scheme (DSSC) able to handle non-convex regularized expected risk minimization. The method operates by decomposing the decision variable into blocks and operating on random subsets of blocks at each step (fusing the merits of stochastic approximation with block coordinate methods), and then implements successive convex approximation. In contrast to many stochastic convex methods whose almost sure behavior is not guaranteed in non-convex settings, DSSC attains almost sure convergence to a stationary solution of the problem. Moreover, we show that the proposed DSSC algorithm achieves stationarity at a rate of ${\mathcal O}((\log t)/{t^{1/4}})$. Numerical experiments on a non-convex variant of a lasso regression problem show that DSSC performs favorably in this setting. We then apply this method to the task of dictionary learning from high-dimensional visual data collected from a ground robot, and observe reliable convergence behavior for a difficult non-convex stochastic program.

中文翻译：

---

双随机连续凸逼近的高维非凸随机优化

在本文中，我们考虑了训练集上的监督学习问题，其中训练样本的数量和特征向量的维度都很大。我们专注于定义我们希望估计的参数质量的损失函数可能是非凸的，但也有凸正则化的情况。我们提出了一种能够处理非凸正则化预期风险最小化的双随机连续凸逼近方案（DSSC）。该方法通过将决策变量分解成块并在每一步对块的随机子集进行操作（融合随机近似与块坐标方法的优点），然后实现逐次凸近似。与许多随机凸方法相比，在非凸设置中不能保证几乎确定的行为，DSSC 几乎可以肯定地收敛到问题的平稳解。此外，我们表明所提出的 DSSC 算法以 ${\mathcal O}((\log t)/{t^{1/4}})$ 的速率实现平稳性。对套索回归问题的非凸变体的数值实验表明 DSSC 在这种情况下表现良好。然后，我们将此方法应用于从地面机器人收集的高维视觉数据中进行字典学习的任务，并观察困难的非凸随机程序的可靠收敛行为。