The use and drawbacks of risk-grouping in prediction models.,BMC Medicine

当前位置： X-MOL 学术 › BMC Med. › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

The use and drawbacks of risk-grouping in prediction models.
BMC Medicine ( IF 9.3 ) Pub Date : 2020-02-03 , DOI: 10.1186/s12916-019-1485-4
Kay See Tan ₁ , Melissa Assel ₁

Affiliation

The goal to provide patients with accurate prognosis has motivated the development of prediction models across different diseases. In renal cell carcinoma (RCC), various prognostic models have been published to guide patient care and facilitate selection into clinical trials, such as the University of California Los Angeles Integrated Scoring System (UISS) [1] to predict overall survival, and the Leibovich model [2] to predict disease-free survival.

Recently, Klatte et al. [3] developed the VENUSS model to predict outcomes following curative surgery for non-metastatic RCC. Unlike the UISS and Leibovich models, the VENUSS model predicts recurrence among papillary RCC, and appropriately utilized a competing risk approach [4] to account for competing events (e.g., deaths without recurrence) in the analyses. The final multivariable model was converted into a simplified scoring algorithm, and then, based on cumulative incidence of recurrence, further categorized into low, intermediate and high-risk groups. The authors then evaluated the predictive performance of the risk groups: after estimating a new model based on dummy variables for each risk group, multiple predictive performance metrics (c-index, calibration plots, decision curve analysis) were assessed. The authors concluded that the VENUSS model may be superior to standard models.

In their study, Klatte et al. [3] demonstrated the clinical importance of VENUSS risk groups to define eligibility in clinical trials. However, when assessing individual patient risks, we argue that the perceived benefit of a user-friendly risk-grouping approach is outweighed by the loss of precision in risk estimation, particularly in the era of personalized medicine.

Risk-grouping provides a qualitative assessment of prognosis by identifying patients at different risk levels for an event of interest. Risk-grouping can also provide a crude estimate of risk using simple back-of-the-envelope calculations, and have thus gained popularity in clinical practice. In the VENUSS model [3], simplified risk scores (0–11 points) are first derived by summing integer points assigned to each level of five clinical characteristics found to be associated with recurrence. Based on cumulative incidence of recurrence curves, the authors then grouped the scores to define low (0–2 points), intermediate (3–5 points) and high (≥6 points) risk groups, corresponding to 5-year cumulative incidence of recurrence of 2.9, 15.4 and 54.5%, respectively. Physicians can thus utilize VENUSS risk groups for prognostic stratification in adjuvant trials.

Risk-grouping leads to loss of information

Categorizing predictions into risk groups implies that the risks (or probabilities) are identical for all individuals within each group, resulting in the loss of granularity in risk estimates. For example, the 5-year cumulative incidence of recurrence in the ‘intermediate risk’ group may range between 10 and 25%, depending on VENUSS scores of 3, 4, or 5 [3]. This crude grouping results in a loss of information crucial for individualized disease management [4].

Precise risk estimation can guide personalized treatment

A clear benefit to prediction models is that patient-specific risk predictions can be directly obtained to guide patient care. Informed treatment decision-making requires the understanding of a patient’s ‘threshold probability’ – the critical point at which the expected benefit of the treatment equals the expected benefit of avoiding the treatment – and above which would prompt a patient to opt for adjuvant treatments. A cancer-averse patient may opt for adjuvant treatment at a predicted 5-year recurrence rate of 5%, whereas a treatment-averse patient may only do so when the risk of recurrence is above 35%. Using a decision curve, physicians can demonstrate the net benefit of receiving adjuvant treatment at various threshold probabilities [5].

The VENUSS study [3] presented multiple decision curves, but their utility in providing patient-specific risks is limited because such risk-grouping de-emphasizes the variability in threshold probabilities. The three predefined risk groups produced exactly three discrete points instead of a continuous curve reflecting a range of potential threshold probabilities. Consider a scenario in which a patient contemplates whether to undergo adjuvant treatment, where an applicable threshold probability for that decision ranges between 10 and 20% for the outcome of recurrence at 5 years: all patients in the VENUSS intermediate risk group (group-based risk of 15.4% at 5 years) would have been recommended for adjuvant treatment. However, depending on where they fall within the risk group, patients may have made a different decision if they were provided with a specific recurrence probability at a landmark time instead of the VENUSS group. Thus, the precision of risk predictions enhances the shared decision-making process between patients and physicians to incorporate individual risk tolerance.

Generating precise risk estimates in the modern era

Instead of risk groups, the predicted probability of recurrence at a clinically relevant timepoint should be utilized for individualized patient care. The latter is more accurate, and can be derived directly from the prediction models. Previously, simplified scoring algorithms were favored because it was tedious and complicated to estimate precise outcome probabilities for time-to-event outcomes. This challenge has now been overcome by technology: prediction models can be translated into nomograms for publication [6], or transformed into web-based calculators [7]. By inputting specific patient characteristics, these open-access prediction tools can provide patient-specific predictions of cancer outcomes across different diseases, such as the 5-year recurrence-free probability following surgery for RCC [7].

The current VENUSS risk grouping is valuable to define cohorts for clinical studies; however, to use VENUSS in the context of estimating patient-specific risk, the following recommendations must be considered. First, following the TRIPOD guidelines [8], the VENUSS study should provide adequate detail (cumulative baseline hazards, nomograms or web-based calculators) to allow calculations of patient-specific risks rather than only group-based risks. Second, any simplification of a developed prediction model is susceptible to some loss of predictive accuracy because of rounding [9]: we recommend formal validation of the VENUSS model using original model regression coefficients and thorough reporting of the predictive performance metrics before and after simplification of the scoring system [8, 10]. Third, comparisons with other RCC models must be conducted on the basis of validating the original model coefficients rather than risk groups. Addressing these recommendations would establish the validity of the VENUSS model for patient-specific risk estimation.

Not applicable.

1.
Zisman A, Pantuck AJ, Dorey F, Chao DH, Gitlitz BJ, Moldawer N, et al. Mathematical model to predict individual survival for patients with renal cell carcinoma. J Clin Oncol. 2002;20:1368–74.
- Article
- Google Scholar
2.
Leibovich BC, Lohse CM, Cheville JC, Zaid HB, Boorjian SA, Frank I, et al. Predicting oncologic outcomes in renal cell carcinoma after surgery. Eur Urol. 2018;73:772–80.
- Article
- Google Scholar
3.
Klatte T, Gallagher KM, Afferi L, Volpe A, Kroeger N, Ribback S, et al. The VENUSS prognostic model to predict disease recurrence following surgery for non-metastatic papillary renal cell carcinoma: development and evaluation using the ASSURE prospective clinical trial cohort. BMC Med. 2019;17:182.
- Article
- Google Scholar
4.
Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. New York: Springer; 2015.
- Google Scholar
5.
Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008;8:53.
- Article
- Google Scholar
6.
Harrell FE, Jr. rms: Regression Modeling Strategies. R package version 5.1–3.1. 2019. https://CRAN.R-project.org/package=rms. Accessed 05 Nov 2019.
- Google Scholar
7.
Memorial Sloan Kettering Cancer Center. Prediction tools/kidney cancer nomograms. Risk of recurrence following surgery. 2019. https://www.mskcc.org/nomograms/renal/post_op. Accessed 05 Nov 2019.
- Google Scholar
8.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 2015;13:1.
- Article
- Google Scholar
9.
Cole TJ, Algorithm AS. 281: scaling and rounding regression coefficients to integers. J R Stat Soc C Appl. 1993;42:261–8.
- Google Scholar
10.
Wolbers M, Koller MT, Witteman JC, Steyerberg EW. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology. 2009;20:555–61.
- Article
- Google Scholar

Download references

Not applicable.

This work was funded by the National Cancer Institute, grant number P30 CA008748. The funder played no role in the development or writing of this manuscript.

Affiliations

Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, 485 Lexington Avenue, 2nd Floor, New York, NY, USA
- Kay See Tan
- & Melissa Assel

Authors

Search for Kay See Tan in:
- PubMed •
- Google Scholar
Search for Melissa Assel in:
- PubMed •
- Google Scholar

Contributions

KST and MA contributed equally to this commentary. Both authors were involved in the development and writing of this manuscript and approved the final version of the manuscript.

Corresponding author

Correspondence to Kay See Tan.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

Cite this article

Tan, K.S., Assel, M. The use and drawbacks of risk-grouping in prediction models. BMC Med 18, 10 (2020). https://doi.org/10.1186/s12916-019-1485-4

Download citation

Received: 19 December 2019
Accepted: 19 December 2019
Published: 03 February 2020
DOI: https://doi.org/10.1186/s12916-019-1485-4

Keywords

Prognostic model
Validation
Clinical trials
Competing risks
Risk categorization
Decision curve analysis
Nomogram

中文翻译：

风险分组在预测模型中的用途和缺点。

为患者提供准确预后的目标已经激发了跨不同疾病的预测模型的开发。在肾细胞癌（RCC）中，已经发布了各种预后模型来指导患者护理并促进选择临床试验，例如，加州大学洛杉矶分校综合评分系统（UISS）[1]可以预测总体生存率，以及Leibovich模型[2]预测无病生存期。

最近，Klatte等人。[3]开发了VENUSS模型来预测非转移性RCC根治性手术后的结果。与UISS和Leibovich模型不同，VENUSS模型可预测乳头状RCC之间的复发，并在分析中适当使用竞争风险方法[4]来解释竞争事件（例如，死亡而无复发）。将最终的多变量模型转换为简化的评分算法，然后基于累积的复发率将其进一步分类为低，中和高风险组。然后，作者评估了风险组的预测性能：在基于每个风险组的虚拟变量估计新模型之后，评估了多个预测性能指标（c指数，校准图，决策曲线分析）。

在他们的研究中，Klatte等人。[3]证明了VENUSS风险组在临床试验中定义资格的临床重要性。但是，在评估单个患者的风险时，我们认为，风险评估的精确性丧失远远超过了用户友好的风险分组方法的感知收益，尤其是在个性化医疗时代。

风险分组通过识别感兴趣事件的不同风险水平的患者，对预后进行定性评估。风险分组还可以使用简单的后包计算提供粗略的风险估计，因此已在临床实践中得到普及。在VENUSS模型[3]中，首先通过将分配给与复发相关的五个临床特征的每个级别的整数点相加得出简化的风险评分（0-11点）。基于复发曲线的累积发生率，作者然后将分数分组以定义低（0–2点），中（3–5点）和高（≥6点）风险组，对应于5年累积复发率分别为2.9、15.4和54.5％。

风险分组导致信息丢失

将预测分类为风险组意味着每个组中所有个体的风险（或概率）都相同，从而导致风险估计的粒度丢失。例如，“中级风险”组的5年累积复发率可能在10％到25％之间，具体取决于VENUSS评分为3、4或5 [3]。这种粗略的分组导致信息丢失对于个体化疾病管理至关重要[4]。

精确的风险估算可以指导个性化治疗

预测模型的明显好处是可以直接获得针对患者的风险预测，以指导患者护理。明智的治疗决策需要了解患者的“阈值概率”（临界点，在该临界点上，治疗的预期收益等于避免治疗的预期收益），并在此之上提示患者选择辅助治疗。厌恶癌症的患者可以选择以预期的5年复发率5％进行辅助治疗，而厌恶癌症的患者只有在复发风险高于35％时才可以选择。使用决策曲线，医生可以证明在各种阈值概率下接受辅助治疗的净收益[5]。

VENUSS研究[3]提出了多个决策曲线，但是它们在提供针对特定患者的风险方面的效用是有限的，因为这种风险分组法不再强调阈值概率的可变性。这三个预定义的风险组产生的恰好是三个离散点，而不是反映一系列潜在阈值概率的连续曲线。考虑一种情况，在这种情况下，患者考虑是否接受辅助治疗，对于5年后的复发，该决定的适用阈值概率在10％到20％之间：VENUSS中危组中的所有患者（基于组的风险）（建议在5年内维持15.4％的机率）进行辅助治疗。但是，根据它们属于风险组的位置，如果患者在标志性时间而非VENUSS组获得了特定的复发概率，他们可能会做出不同的决定。因此，风险预测的准确性增强了患者和医生之间共享个体风险承受能力的共同决策过程。

在现代时代生成精确的风险估算

代替风险组，应将在临床相关时间点的预测复发概率用于个体化患者护理。后者更为准确，可以直接从预测模型中得出。以前，简化计分算法是受青睐的，因为估计事件到时间结果的精确结果概率既繁琐又复杂。现在，这一挑战已被技术克服：预测模型可以转换为用于发布的列线图[6]，或转换为基于Web的计算器[7]。通过输入特定的患者特征，这些开放获取预测工具可以为不同疾病的癌症预后提供患者特定的预测，例如RCC手术后5年无复发的可能性[7]。

当前的VENUSS风险分组对于定义临床研究队列非常有价值。但是，要在估计患者特定风险的情况下使用VENUSS，必须考虑以下建议。首先，遵循TRIPOD指南[8]，VENUSS研究应提供足够的详细信息（累积基准风险，列线图或基于网络的计算器），以便计算患者特定风险，而不仅仅是基于群体的风险。其次，由于四舍五入，对已开发的预测模型进行的任何简化都可能导致预测准确性的损失[9]：我们建议使用原始模型回归系数对VENUSS模型进行正式验证，并在简化和简化之前和之后全面报告预测性能指标计分系统[8，10]。第三，与其他RCC模型的比较必须在验证原始模型系数而不是风险组的基础上进行。解决这些建议将建立VENUSS模型对特定患者风险评估的有效性。

不适用。

1。
Zisman A，Pantuck AJ，Dorey F，Chao DH，Gitlitz BJ，Moldawer N等。预测肾细胞癌患者个体生存的数学模型。J临床Oncol。2002; 20：1368-74。
- 文章
- 谷歌学术
2。
Leibovich BC，Lohse CM，Cheville JC，Zaid HB，Boorjian SA，Frank I等。预测术后肾细胞癌的肿瘤学结局。Eur Urol。2018; 73：772–80。
- 文章
- 谷歌学术
3。
Klatte T，Gallagher KM，Afferi L，Volpe A，Kroeger N，Ribback S等。VENUSS预后模型可预测非转移性乳头状肾细胞癌手术后的疾病复发：使用ASSURE前瞻性临床试验队列进行开发和评估。BMC Med。2019; 17：182。
- 文章
- 谷歌学术
4。
Harrell FE Jr.回归建模策略：应用于线性模型，逻辑和序数回归以及生存分析。纽约：施普林格；2015年。
- 谷歌学术
5，
Vickers AJ，Cronin AM，Elkin EB，GonenM。扩展到决策曲线分析，这是一种评估诊断测试，预测模型和分子标记的新方法。BMC Med通知Decis Mak。2008; 8：53。
- 文章
- 谷歌学术
6。
Harrell FE，Jr. rms：回归建模策略。R软件包版本5.1–3.1。2019.https：//CRAN.R-project.org/package=rms。于2019年11月5日访问。
- 谷歌学术
7。
斯隆·凯特琳纪念癌症中心。预测工具/肾脏癌列线图。手术后复发的风险。2019.https：//www.mskcc.org/nomograms/renal/post_op。于2019年11月5日访问。
- 谷歌学术
8。
Collins GS，Reitsma JB，Altman DG和Moons KG。透明地报告用于个体预后或诊断的多变量预测模型（TRIPOD）：TRIPOD语句。BMC Med。2015; 13：1。
- 文章
- 谷歌学术
9。
Cole TJ，算法AS。281：将回归系数缩放并四舍五入为整数。JR Stat Soc C Appl。1993; 42：261-8。
- 谷歌学术
10。
Wolbers M，Koller MT，Witteman JC，Steeerberg EW。具有竞争风险的预后模型：方法和在冠状动脉风险预测中的应用。流行病学。2009; 20：555–61。
- 文章
- 谷歌学术

下载参考

不适用。

这项工作是由美国国家癌症研究所资助的，批准号为P30 CA008748。出资者在本手稿的开发或撰写中没有任何作用。

隶属关系

美国纽约州列克星敦大街485号斯隆·凯特琳纪念癌症中心流行病学和生物统计学系
- 凯西谭
- ＆梅利莎·阿瑟（Melissa Assel）

在以下位置搜索Kay See Tan：
- 考研•
- 谷歌学术
在以下位置搜索Melissa Assel：
- 考研•
- 谷歌学术

会费

KST和MA对这一评论做出了同样的贡献。两位作者都参与了该手稿的开发和编写，并批准了该手稿的最终版本。

通讯作者

对应于凯西·谭。

道德规范的批准和同意参加

不适用。

同意发表

不适用。

利益争夺

作者宣称他们没有竞争利益。

发行人注意

对于出版的地图和机构隶属关系中的管辖权主张，Springer Nature保持中立。

开放获取本文根据知识共享署名4.0国际许可（http://creativecommons.org/licenses/by/4.0/）的条款进行分发，该许可允许您以任何方式在任何介质中进行无限制的使用，分发和复制。适当的版权归原始作者和来源，提供指向知识共享许可的链接，并指出是否进行了更改。除非另有说明，否则知识共享公共领域专用豁免（http://creativecommons.org/publicdomain/zero/1.0/）适用于本文提供的数据。

转载和许可