Building trustworthy and transparent image-based medical artificial intelligence (AI) systems requires the ability to interrogate data and models at all stages of the development pipeline, from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. In the present study, we present a foundation model approach, named MONET (medical concept retriever), which learns how to connect medical images with text and densely scores images on concept presence to enable important tasks in medical AI development and deployment such as data auditing, model auditing and model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones and imaging modalities. We trained MONET based on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, competitively with supervised models built on previously concept-annotated dermatology datasets of clinical images. We demonstrate how MONET enables AI transparency across the entire AI system development pipeline, from building inherently interpretable models to dataset and model auditing, including a case study dissecting the results of an AI clinical trial.

构建值得信赖且透明的基于图像的医疗人工智能 (AI) 系统需要能够在开发流程的各个阶段（从训练模型到部署后监控）询问数据和模型。理想情况下，数据和相关的人工智能系统可以使用医生已经熟悉的术语来描述，但这需要用语义上有意义的概念密集注释的医疗数据集。在本研究中，我们提出了一种名为 MONET（医学概念检索器）的基础模型方法，它学习如何将医学图像与文本连接起来，并对概念存在的图像进行密集评分，以实现医疗人工智能开发和部署中的重要任务，例如数据审计、模型审核和模型解释。由于疾病、肤色和成像方式的异质性，皮肤科为 MONET 的多功能性提供了要求严格的用例。我们根据 105,550 张皮肤病图像以及大量医学文献中的自然语言描述来训练 MONET。 MONET 可以准确注释经委员会认证的皮肤科医生验证的皮肤病学图像中的概念，与基于先前概念注释的临床图像皮肤病学数据集构建的监督模型竞争。我们展示了 MONET 如何在整个人工智能系统开发流程中实现人工智能透明度，从构建本质上可解释的模型到数据集和模型审计，包括剖析人工智能临床试验结果的案例研究。