Abstract–We use a connection between compositional kernels and branching processes via Mehler’s formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases. We investigate the memorization capacity of the compositional kernels and neural networks by characterizing the interplay among compositional depth, sample size, dimensionality, and nonlinearity of the activation. Explicit formulas on the eigenvalues of the compositional kernel are provided, which quantify the complexity of the corresponding reproducing kernel Hilbert space. On the methodological front, we propose a new random features algorithm, which compresses the compositional layers by devising a new activation function. Supplementary materials for this article are available online.

摘要-我们通过 Mehler 公式使用组合内核和分支过程之间的联系来研究深度神经网络。这种新的概率见解为我们提供了关于激活函数在组合神经网络中的数学作用的新视角。我们研究了成分内核的未缩放和重新缩放限制，并随着成分深度的增加探索限制行为的不同阶段。我们通过表征成分深度、样本大小、维度和激活的非线性之间的相互作用来研究成分核和神经网络的记忆能力。给出了组合核特征值的显式公式，量化了对应的再生核希尔伯特空间的复杂度。在方法论方面，我们提出了一种新的随机特征算法，它通过设计一个新的激活函数来压缩组合层。本文的补充材料可在线获取。