Deep neural networks (DNN) have shown remarkable success in a variety of machine learning applications. The capacity of these models (i.e., number of parameters), endows them with expressive power and allows them to reach the desired performance. In recent years, there is an increasing interest in deploying DNNs to resource-constrained devices (i.e., mobile devices) with limited energy, memory, and computational budget. To address this problem, we propose Entropy-Constrained Trained Ternarization (EC2T), a general framework to create sparse and ternary neural networks which are efficient in terms of storage (e.g., at most two binary-masks and two full-precision values are required to save a weight matrix) and computation (e.g., MAC operations are reduced to a few accumulations plus two multiplications). This approach consists of two steps. First, a super-network is created by scaling the dimensions of a pre-trained model (i.e., its width and depth). Subsequently, this super-network is simultaneously pruned (using an entropy constraint) and quantized (that is, ternary values are assigned layer-wise) in a training process, resulting in a sparse and ternary network representation. We validate the proposed approach in CIFAR-10, CIFAR-100, and ImageNet datasets, showing its effectiveness in image classification tasks.

中文翻译：

---

使用熵约束训练三元化 (EC2T) 学习稀疏和三元神经网络

深度神经网络 (DNN) 在各种机器学习应用中取得了显着的成功。这些模型的容量（即参数的数量）赋予它们表达能力并允许它们达到所需的性能。近年来，人们越来越关注将 DNN 部署到能源、内存和计算预算有限的资源受限设备（即移动设备）。为了解决这个问题，我们提出了熵约束训练三元化 (EC2T)，这是一个创建稀疏和三元神经网络的通用框架，这些网络在存储方面是有效的（例如，最多需要两个二进制掩码和两个全精度值）以保存权重矩阵）和计算（例如，MAC 操作减少到几个累加加两个乘法）。这种方法包括两个步骤。第一的，超级网络是通过缩放预训练模型的维度（即，其宽度和深度）来创建的。随后，这个超级网络在训练过程中同时被修剪（使用熵约束）和量化（即，三元值被逐层分配），从而产生稀疏和三元网络表示。我们在 CIFAR-10、CIFAR-100 和 ImageNet 数据集中验证了所提出的方法，显示了其在图像分类任务中的有效性。