Background:There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated.Methods:A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.Results:We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P<0.001).Conclusions:Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

中文翻译：

---

心血管临床预测模型的外部验证：大规模文献回顾

背景：有许多临床预测模型 (CPM) 可用于为心血管疾病患者的治疗决策提供信息。然而，它们经过外部测试的程度以及它们的总体表现尚未得到广泛评估。 方法：2017 年 3 月 22 日运行了 SCOPUS 引文搜索，以识别 Tufts Predictive Analytics 中心血管 CPM 的外部验证和比较有效性 CPM 注册。我们评估了外部验证的程度、跨数据库的性能异质性，并探索了与模型性能相关的因素，包括对推导和验证数据之间临床相关性的全局评估。结果：我们确定了 1382 个 CPM 的 2030 年外部验证。注册表中八百七 (58%) 个 CPM 从未经过外部验证。平均而言，每个 CPM 有 1.5 个验证（范围，0-94）。接受者操作特征曲线下的中位数外部验证区域为 0.73（第 25-75 个百分位 [四分位距 (IQR)]，0.66-0.79），代表辨别力下降的中位数百分比为 -11.1%（IQR，-32.4% 至 + 2.7%）与推导数据的性能相比。接受者操作特征曲线下 81% (n=1333) 的验证报告区域显示出低于推导数据集中报告的区分。53% (n=983) 的验证报告了 CPM 校准的一些测量。对于多次评估的 CPM，通常会有很大的性能范围。在按相关性分类的 1702 个验证中，歧视的百分比变化为 -3。P <0.001）。结论：许多已发表的心血管 CPM 从未经过外部验证，对于那些已在开发过程中表现出明显表现的人来说，通常过于乐观。单一的外部验证似乎不足以广泛了解不同设置之间的性能异质性。