Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dynamical systems have been instrumental in demystifying acceleration phenomena in optimization. Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyze the symplectic structure of both Nesterov and heavy ball, besides providing several new insights into these methods. Moreover, we propose a new algorithm based on a dissipative relativistic system that normalizes the momentum and may result in more stable/faster optimization. Importantly, such a method generalizes both Nesterov and heavy ball, each being recovered as distinct limiting cases, and has potential advantages at no additional cost.

中文翻译：

---

保形辛和相对论优化

可以说，机器学习中两种最流行的加速或基于动量的优化方法是 Nesterov 的加速梯度和 Polyaks 的重球，两者都对应于具有摩擦的特定二阶微分方程的不同离散化。这种与连续时间动力系统的联系有助于揭开优化中的加速现象的神秘面纱。在这里，我们研究了某一类耗散（共形）哈密顿系统的结构保持离散化，使我们能够分析 Nesterov 和重球的辛结构，此外还提供了对这些方法的一些新见解。此外，我们提出了一种基于耗散相对论系统的新算法，该算法将动量归一化，并可能导致更稳定/更快的优化。重要的，