SIAM Journal on Scientific Computing, Volume 43, Issue 5, Page A3055-A3081, January 2021.
The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50--100$\times$ across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs.

中文翻译：

---

理解和减轻物理信息神经网络中的梯度流病理

SIAM 科学计算杂志，第 43 卷，第 5 期，第 A3055-A3081 页，2021 年 1 月。
神经网络在不同科学领域的广泛使用通常涉及约束它们以满足某些对称性、守恒定律或其他领域知识。这些约束通常在模型训练期间作为软惩罚施加，并有效地充当经验风险损失的特定领域正则化器。物理信息神经网络是这种哲学的一个例子，其中深度神经网络的输出被约束为近似满足一组给定的偏微分方程。在这项工作中，我们回顾了科学机器学习的最新进展，特别关注物理信息神经网络在预测物理系统结果和从嘈杂数据中发现隐藏物理的有效性。我们还识别并分析了此类方法的基本故障模式，该模式与导致模型训练期间不平衡的反向传播梯度的数值刚度有关。为了解决这个限制，我们提出了一种学习率退火算法，该算法在模型训练期间利用梯度统计来平衡复合损失函数中不同项之间的相互作用。我们还提出了一种新的神经网络架构，它对这种梯度病理更具弹性。总而言之，我们的发展为约束神经网络的训练提供了新的见解，并在计算物理中的一系列问题中持续将物理信息神经网络的预测精度提高了 50--100$\times$。本手稿随附的所有代码和数据均可在 https 上公开获取：