Multimodal learning usually requires a complete set of modalities during inference to maintain performance. Although training data can be well-prepared with high-quality multiple modalities, in many cases of clinical practice, only one modality can be acquired and important clinical evaluations have to be made based on the limited single modality information. In this work, we propose a privileged knowledge learning framework with the ‘Teacher-Student’ architecture, in which the complete multimodal knowledge that is only available in the training data (called privileged information) is transferred from a multimodal teacher network to a unimodal student network, via both a pixel-level and an image-level distillation scheme. Specifically, for the pixel-level distillation, we introduce a regularized knowledge distillation loss which encourages the student to mimic the teacher’s softened outputs in a pixel-wise manner and incorporates a regularization factor to reduce the effect of incorrect predictions from the teacher. For the image-level distillation, we propose a contrastive knowledge distillation loss which encodes image-level structured information to enrich the knowledge encoding in combination with the pixel-level distillation. We extensively evaluate our method on two different multi-class segmentation tasks, i.e., cardiac substructure segmentation and brain tumor segmentation. Experimental results on both tasks demonstrate that our privileged knowledge learning is effective in improving unimodal segmentation and outperforms previous methods.

中文翻译：

---

使用特权多模态知识学习单模态分割

多模态学习在推理过程中通常需要一套完整的模态来保持性能。虽然训练数据可以通过高质量的多模态来准备，但在许多临床实践中，只能获得一种模态，并且必须根据有限的单模态信息进行重要的临床评估。在这项工作中，我们提出了一个具有“教师-学生”架构的特权知识学习框架，其中仅在训练数据中可用的完整多模态知识（称为特权信息）从多模态教师网络转移到单模态学生网络，通过像素级和图像级蒸馏方案。具体来说，对于像素级蒸馏，我们引入了一个正则化的知识蒸馏损失，它鼓励学生以像素方式模仿教师的软化输出，并结合一个正则化因子来减少教师不正确预测的影响。对于图像级蒸馏，我们提出了一种对比知识蒸馏损失，它对图像级结构化信息进行编码，以结合像素级蒸馏来丰富知识编码。我们在两个不同的多类分割任务上广泛评估我们的方法，即心脏子结构分割和脑肿瘤分割。这两项任务的实验结果表明，我们的特权知识学习在改进单峰分割方面是有效的，并且优于以前的方法。