Deep learning has revolutionized the field of medical artificial intelligence (AI) for its excellent diagnostic performance in detecting various diseases [1,2]. For example, based on deep learning, glaucomatous optic neuropathy can be automatically identified from fundus images with an area under the receiver operating characteristic curve (AUC) of 0.986 [3], and patients with COVID-19 pneumonia can be accurately identified from patients with other common types of pneumonia and normal controls using computed tomography images with an AUC of 0.971 [4].
Despite the performance of image-based deep learning algorithms in detecting diseases is ideal in laboratory settings, their real-world performance is uncertain because most studies develop and evaluate deep learning models using only eligible images [[3], [4], [5], [6], [7]]. In real-world settings, multiple factors can cause the generation of ineligible images including poor-quality and poor-location images. In ophthalmology, poor-quality fundus images can be resulted from operator errors, patient noncompliance, hardware imperfections, and obscured optical media [8,9]. Poor-location fundus images can be caused by patients’ poor target fixation during fundus imaging, such as the image without showing the optic disk in a glaucoma screening programme [3]. Ineligible fundus images will potentially cause diagnostic information loss and subsequently have a negative impact on downstream analysis [10,11]. To improve diagnostic accuracy, an approach that can detect and filter out ineligible images is essential to ensure that the subsequent analyses are based on eligible images. Manual ineligible image screening often requires experts, and this procedure is time-consuming and labor-intensive, especially in large-scale applications. In addition, it is not appropriate that automated AI diagnostic systems applied in the real world need human experts to manually check the eligibility of images first. Therefore, an automated image eligibility verification method appears crucial.
Several studies have developed deep learning models to identify poor-quality fundus images corresponding to specific diseases [[12], [13], [14]]. However, the generalizability of these models is limited because the image quality standards are disease-specific. For example, for glaucoma screening, a poor-quality image is defined when vessels within the optic disk region cannot be discerned [3], while for identifying age-related macular degeneration (AMD), a poor-quality image is defined if vessels within the macular region cannot be identified or over 50 % of the macular region is obscured [12]. Briefly, the image quality assessment model built for one specific disease may not be adapted to another. In addition, it is worth noting that all previous studies mainly focused on an automated approach to detect poor-quality images. Yet an automated approach to identify poor-location images which also negatively affect the subsequent image analyses is not well investigated. Therefore, the development of a versatile automated image eligibility verification method that can detect and filter out both poor-quality and poor-location fundus images is more practical for the demands in real-world settings, as AI-based fundus imaging will ultimately be employed to screen for multiple fundus diseases simultaneously [15].
The purpose of this study is to develop a deep learning-based image eligibility verification system (DLIEVS) to detect and filter out ineligible fundus images including both poor-quality and poor-location images to enable that the subsequent image analyses can be based on eligible images. In addition, we evaluated the effectiveness and generalizability of this system using images from different types of fundus cameras in 4 different institutions.
The rest of the paper is organized as follows: Section 2 introduces the databases and methods proposed in this study; Sections 3 and 4 present the performance analyses and clinical value of the proposed methods, respectively; finally, Section 5 provides conclusions and future directions.
