The success of deep learning is usually accompanied by the growth in neural network depth. However, the traditional training method only supervises the neural network at its last layer and propagates the supervision layer-by-layer, which leads to hardship in optimizing the intermediate layers. Recently, deep supervision has been proposed to add auxiliary classifiers to the intermediate layers of deep neural networks. By optimizing these auxiliary classifiers with the supervised task loss, the supervision can be applied to the shallow layers directly. However, deep supervision conflicts with the well-known observation that the shallow layers learn low-level features instead of task-biased high-level semantic features. To address this issue, this paper proposes a novel training framework named Contrastive Deep Supervision, which supervises the intermediate layers with augmentation-based contrastive learning. Experimental results on nine popular datasets with eleven models demonstrate its effects on general image classification, fine-grained image classification and object detection in supervised learning, semi-supervised learning and knowledge distillation. Codes have been released in Github.

中文翻译：

---

对比深度监督

深度学习的成功通常伴随着神经网络深度的增长。然而，传统的训练方法只对神经网络的最后一层进行监督，并逐层传播监督，导致中间层优化困难。最近，有人提出深度监督将辅助分类器添加到深度神经网络的中间层。通过使用监督任务损失优化这些辅助分类器，监督可以直接应用于浅层。然而，深度监督与众所周知的观察相冲突，即浅层学习低级特征而不是任务偏向的高级语义特征。为了解决这个问题，本文提出了一种新的训练框架，称为对比深度监督，它通过基于增强的对比学习来监督中间层。在 9 个流行数据集和 11 个模型上的实验结果证明了它在监督学习、半监督学习和知识蒸馏中对一般图像分类、细粒度图像分类和对象检测的影响。代码已在 Github 上发布。