SIAM Journal on Imaging Sciences, Volume 14, Issue 3, Page 913-945, January 2021.
Finding latent structures in data is drawing increasing attention in diverse fields such as image and signal processing, fluid dynamics, and machine learning. In this work we examine the problem of finding the main modes of gradient flows. Gradient descent is a fundamental process in optimization where its stochastic version is prominent in training of neural networks. Here our aim is to establish a consistent theory for gradient flows $\boldsymbol{\psi}_t = P(\boldsymbol{\psi})$, where $P$ is a nonlinear homogeneous operator. Our proposed framework stems from analytic solutions of homogeneous flows, previously formalized by Cohen and Gilboa, where the initial condition $\boldmath{\psi}_0$ admits the nonlinear eigenvalue problem $P(\boldsymbol{\psi}_0)=\lambda \boldsymbol{\psi}_0 $. We first present an analytic solution for dynamic mode decomposition (DMD) in such cases. We show an inherent flaw of DMD, which is unable to recover the essential dynamics of the flow. It is evident that DMD is best suited for homogeneous flows of degree one. We propose an adaptive time sampling scheme and show its dynamics are analogue to homogeneous flows of degree one with a fixed step size. Moreover, we adapt DMD to yield a real spectrum, using symmetric matrices. Our analytic solution of the proposed scheme recovers the dynamics perfectly and yields zero error. We then proceed to show the relation between the orthogonal modes $\{\phi_i\}$ and their decay profiles under the gradient flow. We formulate orthogonal nonlinear spectral decomposition (OrthoNS), which recovers the essential latent structures of the gradient descent process. Definitions for spectrum and filtering are given, and a Parseval-type identity is shown. Experimental results on images show the resemblance to direct computations of nonlinear spectral decomposition. A significant speedup (by about two orders of magnitude) is achieved for this application using the proposed method.

中文翻译：

---

均匀梯度流的模式

SIAM 成像科学杂志，第 14 卷，第 3 期，第 913-945 页，2021 年 1 月。
寻找数据中的潜在结构在图像和信号处理、流体动力学和机器学习等不同领域引起了越来越多的关注。在这项工作中，我们研究了寻找梯度流的主要模式的问题。梯度下降是优化的一个基本过程，它的随机版本在神经网络的训练中很突出。在这里，我们的目标是为梯度流 $\boldsymbol{\psi}_t = P(\boldsymbol{\psi})$ 建立一致的理论，其中 $P$ 是非线性齐次算子。我们提出的框架源于齐次流的解析解，之前由 Cohen 和 Gilboa 形式化，其中初始条件 $\boldmath{\psi}_0$ 承认非线性特征值问题 $P(\boldsymbol{\psi}_0)=\lambda \boldsymbol{\psi}_0 $. 在这种情况下，我们首先提出了动态模式分解 (DMD) 的解析解。我们展示了 DMD 的一个固有缺陷，它无法恢复流动的基本动态。很明显，DMD 最适合一级均匀流。我们提出了一种自适应时间采样方案，并表明其动态类似于具有固定步长的 1 级均匀流。此外，我们使用对称矩阵调整 DMD 以产生真实频谱。我们对所提出方案的解析解完美地恢复了动力学并产生零误差。然后我们继续展示正交模式 $\{\phi_i\}$ 和它们在梯度流下的衰减曲线之间的关系。我们制定了正交非线性谱分解 (OrthoNS)，它恢复了梯度下降过程的基本潜在结构。给出了频谱和过滤的定义，并显示了 Parseval 类型的身份。图像上的实验结果表明与非线性谱分解的直接计算相似。使用所提出的方法，该应用程序实现了显着的加速（大约两个数量级）。