We study the dynamics of information processing in the continuum depth limit of deep feed-forward Neural Networks (NN) and find that it can be described in language similar to the Renormalization Group (RG). The association of concepts to patterns by a NN is analogous to the identification of the few variables that characterize the thermodynamic state obtained by the RG from microstates. To see this, we encode the information about the weights of a NN in a Maxent family of distributions. The location hyper-parameters represent the weights estimates. Bayesian learning of a new example determine new constraints on the generators of the family, yielding a new probability distribution which can be seen as an entropic dynamics of learning, yielding a learning dynamics where the hyper-parameters change along the gradient of the evidence. For a feed-forward architecture the evidence can be written recursively from the evidence up to the previous layer convoluted with an aggregation kernel. The continuum limit leads to a diffusion-like PDE analogous to Wilson’s RG but with an aggregation kernel that depends on the weights of the NN, different from those that integrate out ultraviolet degrees of freedom. This can be recast in the language of dynamical programming with an associated Hamilton–Jacobi–Bellman equation for the evidence, where the control is the set of weights of the neural network.

中文翻译：

---

神经网络中的熵动力学、重整化群和 Hamilton-Jacobi-Bellman 方程

我们研究了深度前馈神经网络 (NN) 的连续深度限制中的信息处理动态，发现它可以用类似于重整化组 (RG) 的语言来描述。NN 将概念与模式关联起来，类似于识别表征 RG 从微观状态获得的热力学状态的少数变量。为了看到这一点，我们在 Maxent 分布族中对有关 NN 权重的信息进行编码。位置超参数表示权重估计。新示例的贝叶斯学习确定了对家庭生成器的新约束，产生了一个新的概率分布，它可以被看作是学习的熵动力学，产生了一个学习动力学，其中超参数沿着证据的梯度变化。对于前馈架构，可以从证据递归地写入证据，直到与聚合内核卷积的前一层。连续体限制导致类似于威尔逊的 RG 的类似扩散的 PDE，但具有依赖于 NN 权重的聚合内核，不同于那些集成紫外线自由度的聚合内核。这可以在动态规划语言中使用相关的 Hamilton-Jacobi-Bellman 方程重新定义，其中控制是神经网络的权重集。不同于那些积分出紫外线自由度的。这可以在动态规划语言中使用相关的 Hamilton-Jacobi-Bellman 方程重新定义，其中控制是神经网络的权重集。不同于那些积分出紫外线自由度的。这可以在动态规划语言中使用相关的 Hamilton-Jacobi-Bellman 方程重新定义，其中控制是神经网络的权重集。