ABSTRACT

Convolutional neural network is an important model in deep learning, where a convolution operation can be represented by a tensor. To avoid exploding/vanishing gradient problems and to improve the generalizability of a neural network, it is desirable to have a convolution operation that nearly preserves the norm, or to have the singular values of the transformation matrix corresponding to the tensor bounded around 1. We propose a penalty function that can constrain the singular values of the transformation matrix to be around 1. We derive an algorithm to carry out the gradient descent minimization of this penalty function in terms of convolution kernel tensors. Numerical examples are presented to demonstrate the effectiveness of the method.

摘要

卷积神经网络是深度学习中的一个重要模型，其中一个卷积运算可以用一个张量来表示。为了避免梯度爆炸/消失问题并提高神经网络的泛化性，我们希望有一个几乎保留范数的卷积运算，或者让与张量相对应的变换矩阵的奇异值在 1 附近。我们提出了一种惩罚函数，可以将变换矩阵的奇异值限制在 1 左右。我们推导出一种算法，根据卷积核张量来执行该惩罚函数的梯度下降最小化。给出了数值例子来证明该方法的有效性。