Abstract The distillation technique helps transform cumbersome neural networks into compact networks so that models can be deployed on alternative hardware devices. The main advantage of distillation-based approaches include a simple training process, supported by most off-the-shelf deep learning software and no special hardware requirements. In this paper, we propose a guideline for distilling the architecture and knowledge of pretrained standard CNNs. The proposed algorithm is first verified on a large-scale task: offline handwritten Chinese text recognition (HCTR). Compared with the CNN in the state-of-the-art system, the reconstructed compact CNN can reduce the computational cost by > 10 × and the model size by > 8 × with negligible accuracy loss. Then, by conducting experiments on two additional classification task datasets: Chinese Text in the Wild (CTW) and MNIST, we demonstrate that the proposed approach can also be successfully applied on mainstream backbone networks.

中文翻译：

---

CNN中用于中文文本识别的联合架构和知识提炼

摘要 蒸馏技术有助于将繁琐的神经网络转换为紧凑的网络，以便模型可以部署在替代硬件设备上。基于蒸馏的方法的主要优点包括简单的训练过程，由大多数现成的深度学习软件支持，并且没有特殊的硬件要求。在本文中，我们提出了提炼预训练标准 CNN 的架构和知识的指南。所提出的算法首先在一个大规模任务上得到验证：离线手写中文文本识别（HCTR）。与最先进系统中的 CNN 相比，重构的紧凑型 CNN 可以将计算成本降低 > 10 倍，模型尺寸降低 > 8 倍，精度损失可以忽略不计。然后，通过对两个额外的分类任务数据集进行实验：