Artificial intelligence (AI) researchers and radiologists have recently reported AI systems that accurately detect COVID-19 in chest radiographs. However, the robustness of these systems remains unclear. Using state-of-the-art techniques in explainable AI, we demonstrate that recent deep learning systems to detect COVID-19 from chest radiographs rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals. We observe that the approach to obtain training data for these AI systems introduces a nearly ideal scenario for AI to learn these spurious ‘shortcuts’. Because this approach to data collection has also been used to obtain training data for the detection of COVID-19 in computed tomography scans and for medical imaging tasks related to other diseases, our study reveals a far-reaching problem in medical-imaging AI. In addition, we show that evaluation of a model on external data is insufficient to ensure AI systems rely on medically relevant pathology, because the undesired ‘shortcuts’ learned by AI systems may not impair performance in new hospitals. These findings demonstrate that explainable AI should be seen as a prerequisite to clinical deployment of machine-learning healthcare models.

人工智能 (AI) 研究人员和放射科医生最近报告了可以准确检测胸部 X 光片中的 COVID-19 的人工智能系统。然而，这些系统的稳健性仍不清楚。在可解释的 AI 中使用最先进的技术，我们证明了最近从胸部 X 光片中检测 COVID-19 的深度学习系统依赖于混杂因素而不是医学病理学，从而造成了系统看似准确但失败的惊人情况在新医院进行测试时。我们观察到，为这些 AI 系统获取训练数据的方法为 AI 学习这些虚假的“捷径”引入了一个近乎理想的场景。由于这种数据收集方法也被用于获取训练数据，以在计算机断层扫描中检测 COVID-19 以及与其他疾病相关的医学成像任务，因此我们的研究揭示了医学成像 AI 中一个影响深远的问题。此外，我们表明，对外部数据模型的评估不足以确保 AI 系统依赖于医学相关的病理学，因为 AI 系统学习的不受欢迎的“捷径”可能不会影响新医院的绩效。这些发现表明，可解释的 AI 应被视为机器学习医疗保健模型临床部署的先决条件。因为人工智能系统学到的不受欢迎的“捷径”可能不会影响新医院的绩效。这些发现表明，可解释的 AI 应被视为机器学习医疗保健模型临床部署的先决条件。因为人工智能系统学到的不受欢迎的“捷径”可能不会影响新医院的绩效。这些发现表明，可解释的 AI 应被视为机器学习医疗保健模型临床部署的先决条件。