Training deep neural networks can be difficult. For classical neural networks, the initialization method by Xavier and Yoshua which is later generalized by He, Zhang, Ren and Sun can facilitate stable training. However, with the recent development of new layer types, we find that the above mentioned initialization methods may fail to lead to successful training. Based on these two methods, we will propose a new initialization by studying the parameter space of a network. Our principal is to put constrains on the growth of parameters in different layers in a consistent way. In order to do so, we introduce a norm to the parameter space and use this norm to measure the growth of parameters. Our new method is suitable for a wide range of layer types, especially for layers with parameter-sharing weight matrices.

中文翻译：

---

深度网络中基于规范统计空间的新初始化方法

训练深度神经网络可能很困难。对于经典神经网络，由Xavier和Yoshua进行的初始化方法（后来由He，Zhang，Ren和Sun推广）可以促进稳定的训练。但是，随着新层类型的最新发展，我们发现上述初始化方法可能无法导致成功的训练。基于这两种方法，我们将通过研究网络的参数空间来提出新的初始化方法。我们的原则是以一致的方式限制不同层中参数的增长。为此，我们向参数空间引入了一个范数，并使用该范数来衡量参数的增长。我们的新方法适用于各种类型的图层，尤其适用于具有参数共享权重矩阵的图层。