A key challenge in training neural networks for a given medical imaging task is the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports are often readily available in medical records and contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. We evaluate our method on three classification tasks and find consistent performance improvements, reducing the need for labeled data by 67%–98%.

中文翻译：

---

对比跨模态预训练：小样本医学成像的通用策略


针对给定医学成像任务训练神经网络的一个关键挑战是难以获得足够数量的手动标记示例。相比之下，文本成像报告通常可以在医疗记录中轻松获得，并且包含专家撰写的丰富但非结构化的解释，作为标准临床实践的一部分。我们建议使用这些文本报告作为弱监督的一种形式，以提高神经网络的图像解释性能，而无需额外的手动标记示例。我们使用图像文本匹配任务来训练特征提取器，然后使用小型标记数据集在监督任务的迁移学习设置中对其进行微调。最终结果是一个神经网络，可以在推理过程中自动解释图像，而不需要文本报告。我们在三个分类任务上评估我们的方法，并发现一致的性能改进，将标记数据的需求减少了 67%–98%。