We perform an analysis of the average generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant “high-dimensional” regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that standard application of theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.

我们对使用梯度下降训练的大型神经网络的平均泛化动力学进行分析。我们研究了与实际相关的“高维”机制，其中网络中的自由参数数量约为或大于数据集中示例的数量。使用随机矩阵理论和线性模型中的精确解，我们得出学习的泛化误差和训练误差动态，并分析它们如何依赖数据的维数和学习问题的信噪比。我们发现，梯度下降学习的动力学自然可以防止大型网络中的过度训练和过度拟合。在自由网络的有效数量等于样本数量的情况下，中间网络规模的过度训练最糟糕大一点。另外，在高维状态下，低泛化误差要求从较小的初始权重开始。然后，我们转向非线性神经网络，并证明使网络非常大不会损害其泛化性能。相反，它实际上可以减少过度训练，即使没有提前停止或进行任何形式的调整也是如此。我们在超完备模型中确定了此行为背后的两种新颖现象：首先，存在权重的冻结子空间，其中梯度下降下没有学习发生；第二，高维状态的统计特性产生条件更好的输入相关性，可以防止过度训练。