Using medical images to evaluate disease severity and change over time is a routine and important task in clinical decision making. Grading systems are often used, but are unreliable as domain experts disagree on disease severity category thresholds. These discrete categories also do not reflect the underlying continuous spectrum of disease severity. To address these issues, we developed a convolutional Siamese neural network approach to evaluate disease severity at single time points and change between longitudinal patient visits on a continuous spectrum. We demonstrate this in two medical imaging domains: retinopathy of prematurity (ROP) in retinal photographs and osteoarthritis in knee radiographs. Our patient cohorts consist of 4861 images from 870 patients in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study and 10,012 images from 3021 patients in the Multicenter Osteoarthritis Study (MOST), both of which feature longitudinal imaging data. Multiple expert clinician raters ranked 100 retinal images and 100 knee radiographs from excluded test sets for severity of ROP and osteoarthritis, respectively. The Siamese neural network output for each image in comparison to a pool of normal reference images correlates with disease severity rank (ρ = 0.87 for ROP and ρ = 0.89 for osteoarthritis), both within and between the clinical grading categories. Thus, this output can represent the continuous spectrum of disease severity at any single time point. The difference in these outputs can be used to show change over time. Alternatively, paired images from the same patient at two time points can be directly compared using the Siamese neural network, resulting in an additional continuous measure of change between images. Importantly, our approach does not require manual localization of the pathology of interest and requires only a binary label for training (same versus different). The location of disease and site of change detected by the algorithm can be visualized using an occlusion sensitivity map-based approach. For a longitudinal binary change detection task, our Siamese neural networks achieve test set receiving operator characteristic area under the curves (AUCs) of up to 0.90 in evaluating ROP or knee osteoarthritis change, depending on the change detection strategy. The overall performance on this binary task is similar compared to a conventional convolutional deep-neural network trained for multi-class classification. Our results demonstrate that convolutional Siamese neural networks can be a powerful tool for evaluating the continuous spectrum of disease severity and change in medical imaging.

使用医学图像来评估疾病的严重程度和随时间的变化是临床决策中的一项常规且重要的任务。分级系统经常被使用，但并不可靠，因为领域专家对疾病严重程度类别阈值存在分歧。这些离散的类别也不能反映疾病严重程度的基本连续范围。为了解决这些问题，我们开发了一种卷积连体神经网络方法来评估单个时间点的疾病严重程度以及连续频谱上纵向患者就诊之间的变化。我们在两个医学成像领域证明了这一点：视网膜照片中的早产儿视网膜病变（ROP）和膝部X光照片中的骨关节炎。我们的患者队列由早产儿视网膜病变影像和信息学 (i-ROP) 队列研究中 870 名患者的 4861 幅图像以及多中心骨关节炎研究 (MOST) 中 3021 名患者的 10,012 幅图像组成，两者均具有纵向成像数据。多位专家临床医生评估者分别对排除的测试集中的 100 张视网膜图像和 100 张膝部 X 光片进行了 ROP 和骨关节炎严重程度的排名。与正常参考图像池相比，每张图像的 Siamese 神经网络输出与疾病严重程度等级相关（ ROP 的ρ = 0.87， 骨关节炎的ρ = 0.89），无论是在临床分级类别内还是在临床分级类别之间。因此，该输出可以代表任何单个时间点疾病严重程度的连续范围。这些输出的差异可用于显示随时间的变化。或者，可以使用 Siamese 神经网络直接比较同一患者在两个时间点的配对图像，从而对图像之间的变化进行额外的连续测量。重要的是，我们的方法不需要手动定位感兴趣的病理学，只需要用于训练的二元标签（相同与不同）。可以使用基于遮挡灵敏度图的方法来可视化算法检测到的疾病位置和变化部位。对于纵向二元变化检测任务，我们的 Siamese 神经网络在评估 ROP 或膝骨关节炎变化时，根据变化检测策略，实现接收算子特征曲线下面积 (AUC) 高达 0.90 的测试集。与针对多类分类训练的传统卷积深度神经网络相比，此二元任务的整体性能相似。我们的结果表明，卷积连体神经网络可以成为评估疾病严重程度和医学成像变化的连续谱的强大工具。