While model compression is increasingly important because of large neural network size, compression-aware training is challenging as it needs sophisticated model modifications and longer training time.In this paper, we introduce regularization frequency (i.e., how often compression is performed during training) as a new regularization technique for a practical and efficient compression-aware training method. For various regularization techniques, such as weight decay and dropout, optimizing the regularization strength is crucial to improve generalization in Deep Neural Networks (DNNs). While model compression also demands the right amount of regularization, the regularization strength incurred by model compression has been controlled only by compression ratio. Throughout various experiments, we show that regularization frequency critically affects the regularization strength of model compression. Combining regularization frequency and compression ratio, the amount of weight updates by model compression per mini-batch can be optimized to achieve the best model accuracy. Modulating regularization frequency is implemented by occasional model compression while conventional compression-aware training is usually performed for every mini-batch.

中文翻译：

---

调制正则化频率以进行有效的压缩感知模型训练

尽管由于大型神经网络的规模而使模型压缩变得越来越重要，但压缩感知训练仍具有挑战性，因为它需要复杂的模型修改和更长的训练时间。在本文中，我们介绍了正则化频率（即训练期间执行压缩的频率）作为一种实用且有效的压缩感知训练方法的新正则化技术。对于各种正则化技术（例如权重衰减和丢失），优化正则化强度对于改善深度神经网络（DNN）的泛化至关重要。虽然模型压缩还需要适当的正则化量，但是仅通过压缩率来控制由模型压缩引起的正则化强度。在各种实验中，我们显示正则化频率严重影响模型压缩的正则化强度。结合正则化频率和压缩率，可以优化每个微型批处理通过模型压缩获得的权重更新量，以实现最佳模型精度。调制正则化频率是通过偶尔的模型压缩来实现的，而常规的压缩感知训练通常是针对每个小批量执行的。