Abstract

Objective

The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).

Materials and Methods

We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.

Results

Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.

Discussion

Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.

Conclusions

An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

中文翻译：

---

用于改善姑息治疗和临终关怀结果的自动化预测模型的预期临床效用：实施前的常规决策分析

摘要

客观的

该研究旨在评估自动化预测模型在增加临终关怀 (EOL) 住院患者的治疗目标讨论 (GOCD) 方面的预期临床效用。

材料和方法

我们从旨在使用自动警报系统增加 EOL 时的 GOCD 的临床医生的角度构建了一个决策模型。替代策略是 4 个预测模型——3 个随机森林模型和修改后的医院一年死亡率风险模型——为具有 1 年死亡率高风险的患者生成警报。他们接受了 2011 年至 2016 年（70 788 名患者）的入院培训，并接受了 2017 年至 2018 年的入院（16 490 名患者）的测试。常规护理中发生的 GOCD 使用代码状态命令进行测量。我们计算了预期风险差异（有警报的有益结果减去在 EOL 时没有警报的有益结果）、需要受益的数量（使收益比常规护理增加 1 个结果所需的警报数量）和净收益（收益）减去成本）每个策略。

结果

模型的 C 统计量介于 0.79 和 0.86 之间。在 EOL 住院的 3773 例（69%）中有 2599 例发生了代码状态命令。在对应于 10% 警报流行率的风险阈值下，预期风险差异范围为 5.4% 到 10.7%，受益所需的数量范围为 5.4 到 10.9 个警报。使用显示的偏好，只有 2 个模型比常规护理提高了净收益。具有诊断预测因子的随机森林模型具有最高的预期值，包括在敏感性分析中。

讨论

具有可接受的预测有效性的预测模型在改进通常决策的能力方面存在显着差异。

结论

建议在验证预测模型后进行临床效用评估，例如使用决策曲线分析，因为模型预测性指标（例如 C 统计量）不能提供临床价值的信息。