当前位置: X-MOL 学术Artif. Intell. Med. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
A new approach for interpretability and reliability in clinical risk prediction: Acute coronary syndrome scenario
Artificial Intelligence in Medicine ( IF 7.5 ) Pub Date : 2021-05-13 , DOI: 10.1016/j.artmed.2021.102113
Francisco Valente 1 , Jorge Henriques 1 , Simão Paredes 2 , Teresa Rocha 2 , Paulo de Carvalho 1 , João Morais 3
Affiliation  

Introduction

The risk prediction of the occurrence of a clinical event is often based on conventional statistical procedures, through the implementation of risk score models. Recently, approaches based on more complex machine learning (ML) methods have been developed. Despite the latter usually have a better predictive performance, they obtain little approval from the physicians, as they lack interpretability and, therefore, clinical confidence. One clinical issue where both types of models have received great attention is the mortality risk prediction after acute coronary syndromes (ACS).

Objective

We intend to create a new risk assessment methodology that combines the best characteristics of both risk score and ML models. More specifically, we aim to develop a method that, besides having a good performance, offers a personalized model and outcome for each patient, presents high interpretability, and incorporates an estimation of the prediction reliability which is not usually available. By combining these features in the same approach we expect that it can boost the confidence of physicians to use such a tool in their daily activity.

Methods

In order to achieve the mentioned goals, a three-step methodology was developed: several rules were created by dichotomizing risk factors; such rules were trained with a machine learning classifier to predict the acceptance degree of each rule (the probability that the rule is correct) for each patient; that information was combined and used to compute the risk of mortality and the reliability of such prediction. The methodology was applied to a dataset of 1111 patients admitted with any type of ACS (myocardial infarction and unstable angina) in two Portuguese hospitals, to assess the 30-days all-cause mortality risk, being validated through a Monte-Carlo cross-validation technique. The performance was compared with state-of-the-art approaches: logistic regression (LR), artificial neural network (ANN), and clinical risk score model (namely the Global Registry of Acute Coronary Events – GRACE).

Results

For the scenario being analyzed, the performance of the proposed approach and the comparison models was assessed through discrimination and calibration. The ability to rank the patients was evaluated through the area under the ROC curve (AUC), and the ability to stratify the patients into low or high-risk groups was determined using the geometric mean (GM) of specificity and sensitivity, the negative predictive value (NPV) and the positive predictive value (PPV). The validation calibration curves were also inspected. The proposed approach (AUC = 81%, GM = 74%, PPV = 17%, NPV = 99%) achieved testing results identical to the standard LR model (AUC = 83%, GM = 73%, PPV = 16%, NPV=99%), but offers superior interpretability and personalization; it also significantly outperforms the GRACE risk model (AUC = 79%, GM = 47%, PPV = 13%, NPV = 98%) and the standard ANN model (AUC = 78%, GM = 70%, PPV = 13%, NPV = 98%). The calibration curve also suggests a very good generalization ability of the obtained model as it approaches the ideal curve (slope = 0.96). Finally, the reliability estimation of individual predictions presented a great correlation with the misclassifications rate.

Conclusion

We developed and described a new tool that showed great potential to guide the clinical staff in the risk assessment and decision-making process, and to obtain their wide acceptance due to its interpretability and reliability estimation properties. The methodology presented a good performance when applied to ACS events, but those properties may have a beneficial application in other clinical scenarios as well.



中文翻译:

临床风险预测中可解释性和可靠性的新方法:急性冠状动脉综合征情景

介绍

临床事件发生的风险预测通常基于常规统计程序,通过实施风险评分模型。最近,已经开发了基于更复杂的机器学习 (ML) 方法的方法。尽管后者通常具有更好的预测性能,但它们几乎没有得到医生的认可,因为它们缺乏可解释性,因此缺乏临床信心。这两种模型都受到极大关注的一个临床问题是急性冠状动脉综合征 (ACS) 后的死亡风险预测。

客观的

我们打算创建一种新的风险评估方法,该方法结合了风险评分和 ML 模型的最佳特征。更具体地说,我们的目标是开发一种方法,除了具有良好的性能外,还为每个患者提供个性化的模型和结果,具有高度的可解释性,并结合了通常不可用的预测可靠性估计。通过在同一方法中结合这些功能,我们期望它可以提高医生在日常活动中使用这种工具的信心。

方法

为了实现上述目标,开发了一种三步法:通过对风险因素进行二分法创建几条规则;这些规则用机器学习分类器进行训练,以预测每个患者对每个规则的接受程度(规则正确的概率);这些信息被结合起来用于计算死亡风险和这种预测的可靠性。该方法应用于葡萄牙两家医院 1111 名患有任何类型 ACS(心肌梗塞和不稳定型心绞痛)的患者的数据集,以评估 30 天全因死亡风险,并通过蒙特卡罗交叉验证进行验证技术。将性能与最先进的方法进行了比较:逻辑回归 (LR)、人工神经网络 (ANN)、

结果

对于正在分析的场景,通过区分和校准来评估所提出的方法和比较模型的性能。通过 ROC 曲线下面积 (AUC) 评估对患者进行排序的能力,并使用特异性和敏感性的几何平均值 (GM) 确定将患者分为低危组或高危组的能力,即阴性预测值值(NPV)和阳性预测值(PPV)。还检查了验证校准曲线。所提出的方法(AUC = 81%,GM = 74%,PPV = 17%,NPV = 99%)获得了与标准 LR 模型相同的测试结果(AUC = 83%,GM = 73%,PPV = 16%,NPV =99%),但提供卓越的可解释性和个性化;它还显着优于 GRACE 风险模型(AUC = 79%,GM = 47%,PPV = 13%,NPV = 98%)和标准 ANN 模型(AUC = 78%,GM = 70%,PPV = 13%,NPV = 98%)。校准曲线还表明获得的模型具有非常好的泛化能力,因为它接近理想曲线(斜率 = 0.96)。最后,个体预测的可靠性估计与误分类率有很大的相关性。

结论

我们开发并描述了一种新工具,该工具在指导临床工作人员进行风险评估和决策过程中显示出巨大潜力,并因其可解释性和可靠性估计特性而获得广泛接受。该方法在应用于 ACS 事件时表现出良好的性能,但这些特性也可能在其他临床场景中具有有益的应用。

更新日期:2021-05-26
down
wechat
bug