We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained optimization and machine learning problems. The algorithm maps the gradient onto a low-dimensional random subspace of dimension \(\ell\) at each iteration, similar to coordinate descent but without restricting directional derivatives to be along the axes. Without requiring a full gradient, this mapping can be performed by computing \(\ell\) directional derivatives (e.g., via forward-mode automatic differentiation). We give proofs for convergence in expectation under various convexity assumptions as well as probabilistic convergence results under strong-convexity. Our method provides a novel extension to the well-known Gaussian smoothing technique to descent in subspaces of dimension greater than one, opening the doors to new analysis of Gaussian smoothing when more than one directional derivative is used at each iteration. We also provide a finite-dimensional variant of a special case of the Johnson–Lindenstrauss lemma. Experimentally, we show that our method compares favorably to coordinate descent, Gaussian smoothing, gradient descent and BFGS (when gradients are calculated via forward-mode automatic differentiation) on problems from the machine learning and shape optimization literature.

我们提出了一种用于无约束优化的随机下降算法，该算法在目标函数评估缓慢且不容易获得梯度的情况下特别有效，例如在某些PDE约束优化和机器学习问题中。该算法在每次迭代时将梯度映射到维\（\ ell \）的低维随机子空间，类似于坐标下降，但不限制方向导数沿轴方向。不需要完整的渐变，可以通过计算\（\ ell \）来执行此映射方向导数（例如，通过前向模式自动微分）。我们给出了各种凸假设下期望收敛的证明，以及强凸条件下的概率收敛结果。我们的方法为众所周知的高斯平滑技术提供了一种新颖的扩展，使其能够在尺寸大于1的子空间中下降，从而在每次迭代中使用多个方向导数时为新的高斯平滑分析打开了大门。我们还提供了Johnson-Lindenstrauss引理的特殊情况的有限维变体。实验表明，我们的方法与协调下降，高斯平滑，