Deep neural networks (DNNs) have achieved physician-level accuracy on many imaging-based medical diagnostic tasks, for example classification of retinal images in ophthalmology. However, their decision mechanisms are often considered impenetrable leading to a lack of trust by clinicians and patients. To alle-viate this issue, a range of explanation methods have been proposed to expose the inner workings of DNNs leading to their decisions. For imaging-based tasks, this is often achieved via saliency maps. The quality of these maps are typically evaluated via perturbation analysis without experts involved. To facilitate the adoption and success of such automated systems, however, it is crucial to validate saliency maps against clinicians. In this study, we used three different network architectures and developed ensembles of DNNs to detect diabetic retinopathy and neovascular age-related macular degeneration from retinal fundus images and optical coherence tomography scans, respectively. We used a variety of explanation methods and obtained a comprehensive set of saliency maps for explaining the ensemble-based diagnostic decisions. Then, we systematically validated saliency maps against clinicians through two main analyses — a direct comparison of saliency maps with the expert annotations of disease-specific pathologies and perturbation analyses using also expert annotations as saliency maps. We found the choice of DNN architecture and explanation method to significantly influence the quality of saliency maps. Guided Backprop showed consistently good performance across disease scenarios and DNN architectures, suggesting that it provides a suitable starting point for explaining the decisions of DNNs on retinal images.

中文翻译：

---

用于理解眼科深度神经网络的显着图的临床验证

深度神经网络 (DNN) 在许多基于成像的医学诊断任务（例如眼科中的视网膜图像分类）上已达到医生级别的准确性。然而，他们的决策机制通常被认为是难以理解的，导致临床医生和患者缺乏信任。为了缓解这个问题，已经提出了一系列解释方法来揭示导致其决策的 DNN 的内部工作原理。对于基于成像的任务，这通常是通过显着图来实现的。这些地图的质量通常通过扰动分析进行评估，而无需专家参与。然而，为了促进此类自动化系统的采用和成功，针对临床医生验证显着图至关重要。在这项研究中，我们使用三种不同的网络架构并开发了 DNN 的集合，分别从视网膜眼底图像和光学相干断层扫描检测糖尿病性视网膜病变和新生血管性年龄相关性黄斑变性。我们使用了多种解释方法并获得了一套全面的显着图来解释基于集成的诊断决策。然后，我们通过两个主要分析系统地验证了针对临床医生的显着图——显着图与特定疾病病理的专家注释的直接比较和使用专家注释作为显着图的扰动分析。我们发现 DNN 架构和解释方法的选择会显着影响显着图的质量。