A hierarchical neural network usually has many singular regions in the parameter space due to the degeneration of hidden units. Here, we focus on a three-layer perceptron, which has one-dimensional singular regions comprising both attractive and repulsive parts. Such a singular region is often called a Milnor-like attractor. It is empirically known that in the vicinity of a Milnor-like attractor, several parameters converge much faster than the rest and that the dynamics can be reduced to smaller-dimensional ones. Here we give a rigorous proof for this phenomenon based on a center manifold theory. As an application, we analyze the reduced dynamics near the Milnor-like attractor and study the stochastic effects of the online learning.

中文翻译：

---

三层感知器退化引起的高原现象的中心流形分析

由于隐藏单元的退化，分层神经网络通常在参数空间中具有许多奇异区域。在这里，我们关注一个三层感知器，它具有一维奇异区域，包括吸引和排斥部分。这样的奇异区域通常被称为类米尔诺吸引子。根据经验可知，在类米尔诺吸引子附近，几个参数的收敛速度比其他参数快得多，并且动力学可以减少到更小维度的参数。在这里，我们基于中心流形理论对这种现象给出了严格的证明。作为一个应用，我们分析了类米尔诺吸引子附近的简化动力学，并研究了在线学习的随机效应。