Any matrix A has an LU decomposition up to a row or column permutation. Less well-known is the fact that it has a ‘Toeplitz decomposition’ $A=T_1{T}_2\cdots T_r$ where $T_i$'s are Toeplitz matrices. We will prove that any continuous function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ has an approximation to arbitrary accuracy by a neural network that maps $x\in {\mathbb{R}}^n$ to $L_1{\sigma }_1{U}_1{\sigma }_2{L}_2{\sigma }_3{U}_2\cdots L_r{\sigma }_{2r-1}{U}_{r}x\in {\mathbb{R}}^m$, i.e., where the weight matrices alternate between lower and upper triangular matrices, ${\sigma }_i(x)≔\sigma (x-b_i)$ for some bias vector $b_i$, and the activation σ may be chosen to be essentially any uniformly continuous nonpolynomial function. The same result also holds with Toeplitz matrices, i.e., $f\approx T_1{\sigma }_1{T}_2{\sigma }_2\cdots {\sigma }_{r-1}{T}_r$ to arbitrary accuracy, and likewise for Hankel matrices. A consequence of our Toeplitz result is a fixed-width universal approximation theorem for convolutional neural networks, which so far have only arbitrary width versions. Since our results apply in particular to the case when f is a general neural network, we may regard them as LU and Toeplitz decompositions of a neural network. The practical implication of our results is that one may vastly reduce the number of weight parameters in a neural network without sacrificing its power of universal approximation. We will present several experiments on real data sets to show that imposing such structures on the weight matrices dramatically reduces the number of training parameters with almost no noticeable effect on test accuracy.

任何矩阵A都具有直到行或列排列的 LU 分解。不太为人所知的是它具有“托普利茨分解”的事实$A={\mathrm{时间}}_1{\mathrm{时间}}_2\cdots {\mathrm{时间}}_r$在哪里${\mathrm{时间}}_{我}$是托普利茨矩阵。我们将证明任意连续函数$F：{\mathbb{右}}^n\to {\mathbb{右}}^{米}$通过映射的神经网络具有任意精度的近似值$X\epsilon {\mathbb{右}}^n$到$L_1{\sigma }_1{U}_1{\sigma }_2{L}_2{\sigma }_3{U}_2\cdots L_r{\sigma }_{2r-1}{U}_{r}X\epsilon {\mathbb{右}}^{米}$，即权重矩阵在下三角矩阵和上三角矩阵之间交替，${\sigma }_{我}（X）≔\sigma （X-{乙}_{我}）$对于一些偏置向量${乙}_{我}$，并且激活σ可以被选择为基本上任何一致连续的非多项式函数。同样的结果也适用于 Toeplitz 矩阵，即$F\approx {\mathrm{时间}}_1{\sigma }_1{\mathrm{时间}}_2{\sigma }_2\cdots {\sigma }_{r-1}{\mathrm{时间}}_r$任意精度，汉克尔矩阵也是如此。我们的托普利茨结果的一个结果是卷积神经网络的固定宽度通用逼近定理，迄今为止只有任意宽度版本。由于我们的结果特别适用于f是一般神经网络的情况，因此我们可以将它们视为神经网络的 LU 和 Toeplitz 分解。我们的结果的实际意义是，可以大大减少神经网络中权重参数的数量，而不牺牲其通用逼近的能力。我们将在真实数据集上进行几个实验，以表明将这种结构应用于权重矩阵可以显着减少训练参数的数量，而对测试精度几乎没有明显影响。