The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model’s performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting “unreliability score” can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability.

识别可能出现不良结果的患者的能力是良好临床护理的重要组成部分。因此，预测风险分层模型在临床决策中发挥着重要作用。确定给定的预测模型是否适合临床使用通常涉及使用成功的标准统计测量（例如准确性、辨别能力）来评估模型在大型患者数据集上的性能。然而，由于这些指标对应于具有一系列不同特征的患者的平均值，因此很难仅使用这些指标来辨别对给定患者的个体预测是否值得信任。在本文中，我们介绍了一种新方法，用于识别预测模型预计较差的患者亚组，从而突出显示给定的预测何时具有误导性且不应被信任。由此产生的“不可靠性评分”可以针对任何临床风险模型进行计算，并且适用于大类别不平衡的情况，这是医疗保健环境中经常遇到的情况。使用全球急性冠脉事件登记处 (GRACE) 中超过 40,000 名患者的数据，我们证明，不可靠性评分较高的患者形成了一个亚组，其中预测模型的准确性和辨别能力均下降。