This paper is concerned with sequential filtering based stochastic optimization (FSO) approaches that leverage a probabilistic perspective to implement the incremental proximity method (IPM). The present FSO methods are derived based on the Kalman filter (KF) and the extended KF (EKF). In contrast with typical methods such as stochastic gradient descent (SGD) and IPMs, they do not need to pre-schedule the learning rate for convergence. Nevertheless, they have limitations that inherit from the KF mechanism. As the particle filtering (PF) method outperforms KF and its variants remarkably for nonlinear non-Gaussian sequential filtering problems, it is natural to ask if FSO methods can benefit from PF to get around of their limitations. We provide an affirmative answer to this question by developing two PF based stochastic optimizers (PFSOs). For performance evaluation, we apply them to address nonlinear least-square fitting with simulated data, and empirical risk minimization for binary classification of real data sets. Experimental results demonstrate that PFSOs outperform remarkably a benchmark SGD algorithm, the vanilla IPM, and KF-type FSO methods in terms of numerical stability, convergence speed, and flexibility in handling diverse types of loss functions.

本文涉及基于顺序过滤的随机优化（FSO）方法，该方法利用概率观点来实现增量邻近方法（IPM）。当前的FSO方法是基于卡尔曼滤波器（KF）和扩展KF（EKF）得出的。与诸如随机梯度下降（SGD）和IPM之类的典型方法相比，它们不需要预先计划学习速率即可收敛。但是，它们具有从KF机制继承的限制。由于在非线性非高斯顺序滤波问题上，粒子滤波（PF）方法的性能优于KF及其变体，因此很自然地要问FSO方法是否可以从PF中受益，从而克服它们的局限性。通过开发两个基于PF的随机优化器（PFSO），我们为这个问题提供了肯定的答案。为了进行性能评估，我们将其用于处理模拟数据的非线性最小二乘拟合，以及对实际数据集进行二值分类的经验风险最小化。实验结果表明，PFSO在数值稳定性，收敛速度和处理各种类型的损失函数的灵活性方面均明显优于基准SGD算法，香草IPM和KF型FSO方法。