Compressed Sensing using \(\ell _1\) regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how to make it work. Following Xiao (J Mach Learn Res 11(Oct):2543–2596, 2010), We first demonstrate that the commonly used stochastic gradient decent and variants training algorithm is not an appropriate match with \(\ell _1\) regularization and then replace it with a different training algorithm based on a regularized dual averaging (RDA) method. The RDA method of Xiao (J Mach Learn Res 11(Oct):2543–2596, 2010) was originally designed specifically for convex problem, but with new theoretical insight and algorithmic modifications (using proper initialization and adaptivity), we have made it an effective match with \(\ell _1\) regularization to achieve a state-of-the-art sparsity for the highly non-convex CNN compared to other weight pruning methods without compromising accuracy (achieving 95% sparsity for ResNet-18 on CIFAR-10, for example).

中文翻译：

---

使$$ \ ell _1 $$ℓ1正则化在训练稀疏CNN时有效

使用\（\ ell _1 \）正则化的压缩感知是许多应用程序中最强大，最流行的稀疏化技术之一，但是为什么不使用它来获得稀疏的深度学习模型（例如卷积神经网络（CNN））呢？本文旨在为这个问题提供答案，并说明如何使其工作。遵循Xiao（J Mach Learn Res 11（Oct）：2543–2596，2010）之后，我们首先证明了常用的随机梯度体面和变异训练算法与\（\ ell _1 \）不合适。正则化，然后将其替换为基于正则化双重平均（RDA）方法的其他训练算法。Xiao的RDA方法（J Mach Learn Res 11（Oct）：2543-2596，2010）最初是专为凸问题设计的，但是由于有了新的理论洞察力和算法修改（使用了适当的初始化和适应性），我们使其成为一种与\（\ ell _1 \）正则表达式的有效匹配，以实现与其他权重修剪方法相比高度非凸CNN的最新稀疏性，而不会影响准确性（CIFAR-上的ResNet-18稀疏性达到95％ 10）。