Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this article, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual , between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04%, 23.14%, 53.59%, and 56.86% of the teachers’ computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.

中文翻译：

---

ResKD：残差引导的知识蒸馏

旨在将知识从沉重的教师网络转移到轻量级学生网络的知识蒸馏已成为一种有前途的压缩神经网络技术。但是，由于重型教师和轻量级学生之间的能力差距，他们之间仍然存在很大的绩效差距。在本文中，我们将利用知识鸿沟或剩余的 ，在老师和学生之间作为指导，以培训一名轻量级的学生（称为再学习生）。我们将学生和转学生合并为一个新学生，转学生纠正了先前学生的错误。可以重复进行这种残差引导的过程，直到用户在准确性和成本之间取得平衡。在推论时，我们提出了一种自适应样本的策略来确定每个样本都不需要哪些再研究生，这可以节省计算成本。实验结果表明，在CIFAR-10，CIFAR-100，Tiny-ImageNet和ImageNet数据集上，我们以教师计算成本的18.04％，23.14％，53.59％和56.86％达到了竞争表现。最后，我们对我们的方法进行了详尽的理论和实证分析。