Medical cross-modal retrieval aims to retrieve semantically similar medical instances across different modalities, such as retrieving X-ray images using radiology reports or retrieving radiology reports using X-ray images. The main challenge for medical cross-modal retrieval are the semantic gap and the small visual differences between different categories of medical images. To address those issues, we present a novel end-to-end deep hashing method, called Deep Medical Cross-Modal Attention Hashing (DMCAH), which extracts the global features utilizing global average pooling and local features by recurrent attention. Specifically, we recursively move from the coarse to fine-grained regions of images to locate discriminative regions more accurately, and recursively extract the discriminative semantic information of texts from the sentence level to the word level. Then, we select the discriminative features by aggregating the finer feature via adaptive attention. Finally, to reduce the semantic gap, we map images and reports features into a common space and obtain the discriminative hash codes. Comprehensive experimental results on large-scale medical dataset MIMIC-CXR and natural scene dataset MS-COCO show that DMCAH can achieve better performance than existing cross-modal hashing methods.

医学跨模态检索旨在跨不同模态检索语义相似的医学实例，例如使用放射学报告检索 X 射线图像或使用 X 射线图像检索放射学报告。医学跨模态检索的主要挑战是不同类别医学图像之间的语义鸿沟和微小的视觉差异。为了解决这些问题，我们提出了一种新颖的端到端深度哈希方法，称为深度医学跨模态注意力哈希（DMCAH），它通过循环注意利用全局平均池化和局部特征提取全局特征。具体来说，我们递归地从图像的粗粒度区域移动到细粒度区域，以更准确地定位判别区域，并从句子级别到词级别递归提取文本的判别语义信息。然后，我们通过自适应注意聚合更精细的特征来选择判别特征。最后，为了减少语义鸿沟，我们将图像和报告特征映射到一个公共空间并获得判别性哈希码。在大规模医学数据集 MIMIC-CXR 和自然场景数据集 MS-COCO 上的综合实验结果表明，DMCAH 可以取得比现有跨模态哈希方法更好的性能。