The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.

中文翻译：

---

神经网络学习中高原现象的数据依赖性——统计力学分析*

各种研究人员已经报告了平台现象，即在学习过程中损失值停止下降。这种现象在 1990 年代被积极检查，发现是由于神经网络模型的基本层次结构。那么这种现象被认为是不可避免的。然而，这种现象在最近的深度学习背景下很少发生。理论与现实之间存在差距。在本文中，我们使用统计力学公式阐明了高原现象与所学习数据的统计特性之间的关系。结果表明，协方差特征值较小且分散的数据往往会使平台现象不明显。