Recurrent neural networks (RNNs) have recently achieved remarkable successes in a number of applications. However, the huge sizes and computational burden of these models make it difficult for their deployment on edge devices. A practically effective approach is to reduce the overall storage and computation costs of RNNs by network pruning techniques. Despite their successful applications, those pruning methods based on Lasso either produce irregular sparse patterns in weight matrices, which is not helpful in practical speedup. To address these issues, we propose a structured pruning method through neuron selection which can remove the independent neuron of RNNs. More specifically, we introduce two sets of binary random variables, which can be interpreted as gates or switches to the input neurons and the hidden neurons, respectively. We demonstrate that the corresponding optimization problem can be addressed by minimizing the L0 norm of the weight matrix. Finally, experimental results on language modeling and machine reading comprehension tasks have indicated the advantages of the proposed method in comparison with state-of-the-art pruning competitors. In particular, nearly 20× practical speedup during inference was achieved without losing performance for the language model on the Penn TreeBank dataset, indicating the promising performance of the proposed method.

中文翻译：

---

通过神经元选择对递归神经网络进行结构化修剪。

递归神经网络（RNN）最近在许多应用中都取得了令人瞩目的成功。但是，这些模型的巨大规模和计算负担使其难以在边缘设备上进行部署。一种实用有效的方法是通过网络修剪技术来减少RNN的总体存储和计算成本。尽管应用成功，那些基于Lasso的修剪方法还是会在重量矩阵中产生不规则的稀疏模式，这在实际加速中无济于事。为了解决这些问题，我们提出了一种通过神经元选择的结构化修剪方法，该方法可以删除RNN的独立神经元。更具体地说，我们引入了两组二进制随机变量，它们可以分别解释为输入神经元和隐藏神经元的门或开关。我们证明，可以通过最小化权重矩阵的L0范数来解决相应的优化问题。最后，关于语言建模和机器阅读理解任务的实验结果表明，与最先进的修剪竞争对手相比，该方法具有优势。特别是，在推理过程中实现了近20倍的实际加速，而不会损失Penn TreeBank数据集上的语言模型的性能，这表明了该方法的良好前景。