We propose a simple yet effective technique to simplify the training and the resulting model of neural networks. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$k elements (in terms of magnitude) are kept. As a result, only $k$k rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction in the computational cost. Based on the sparsified gradients, we further simplify the model by eliminating the rows or columns that are seldom updated, which will reduce the computational cost both in the training and decoding, and potentially accelerate decoding in real-world applications. Surprisingly, experimental results demonstrate that most of the time we only need to update fewer than 5 percent of the weights at each back propagation pass. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The model simplification results show that we could adaptively simplify the model which could often be reduced by around 9x, without any loss on accuracy or even with improved accuracy.

中文翻译：

---

深度学习的训练简化和模型简化：最小努力反向传播方法

我们提出了一种简单而有效的技术来简化神经网络的训练和生成的模型。在反向传播中，只计算完整梯度的一小部分来更新模型参数。梯度向量以这样一种方式稀疏化，即只有顶部-$千$克元素（在数量方面）被保留。结果，只有$千$克权重矩阵的行或列（取决于布局）被修改，导致计算成本的线性降低。基于稀疏梯度，我们通过消除很少更新的行或列来进一步简化模型，这将降低训练和解码中的计算成本，并可能加速实际应用中的解码。令人惊讶的是，实验结果表明，在大多数情况下，我们只需要在每次反向传播时更新不到 5% 的权重。更有趣的是，结果模型的准确率实际上是提高而不是降低，并给出了详细的分析。模型简化结果表明，我们可以自适应地简化模型，通常可以减少约 9 倍，