Clinical visit data are clustered within people, which complicates prediction modeling. Cluster size is often informative because people receiving more care are less healthy and at higher risk of poor outcomes. We used data from seven health systems on 1,518,968 outpatient mental health visits from January 1, 2012 to June 30, 2015 to predict suicide attempt within 90 days. We evaluated true performance of prediction models using a prospective validation set of 4,286,495 visits from October 1, 2015 to September 30, 2017. We examined dividing clustered data on the person or visit level for model training and cross-validation and considered a within cluster resampling approach for model estimation. We evaluated optimism by comparing estimated performance from a left-out testing dataset to performance in the prospective dataset. We used two prediction methods, logistic regression with least absolute shrinkage and selection operator (LASSO) and random forest. The random forest model using a visit-level split for model training and testing was optimistic; it overestimated discrimination (area under the curve, AUC = 0.95 in testing versus 0.84 in prospective validation) and classification accuracy (sensitivity = 0.48 in testing versus 0.19 in prospective validation, 95th percentile cut-off). Logistic regression and random forest models using a person-level split performed well, accurately estimating prospective discrimination and classification: estimated AUCs ranged from 0.85 to 0.87 in testing versus 0.85 in prospective validation, and sensitivity ranged from 0.15 to 0.20 in testing versus 0.17 to 0.19 in prospective validation. Within cluster resampling did not improve performance. We recommend dividing clustered data on the person level, rather than visit level, to ensure strong performance in prospective use and accurate estimation of future performance at the time of model development.

中文翻译：

---

临床风险预测模型和信息集群大小：评估自杀风险预测算法的性能

临床就诊数据集中在人群中，这使预测建模变得复杂。集群规模通常会提供信息，因为接受更多护理的人健康状况较差，并且出现不良结果的风险较高。我们使用了 2012 年 1 月 1 日至 2015 年 6 月 30 日期间来自 7 个卫生系统的 1,518,968 次门诊心理健康访问的数据来预测 90 天内的自杀企图。我们使用从 2015 年 10 月 1 日到 2017 年 9 月 30 日的 4,286,495 次访问的前瞻性验证集评估了预测模型的真实性能。我们检查了在人员或访问级别上划分聚类数据以进行模型训练和交叉验证，并考虑了聚类内重采样模型估计的方法。我们通过比较来自遗漏测试数据集的估计性能与预期数据集中的性能来评估乐观度。我们使用了两种预测方法，具有最小绝对收缩和选择算子 (LASSO) 的逻辑回归和随机森林。使用访问级别拆分进行模型训练和测试的随机森林模型是乐观的；它高估了区分度（曲线下面积，AUC = 0.95 测试与 0.84 前瞻性验证）和分类准确性（灵敏度 = 0.48 测试与前瞻性验证 0.19，95% 截止）。使用人员级别拆分的逻辑回归和随机森林模型表现良好，准确估计了前瞻性歧视和分类：估计的 AUC 在测试中为 0.85 至 0.87，而在前瞻性验证中为 0.85，灵敏度在测试中为 0.15 至 0.20，而测试中为 0.17 至 0.19在前瞻性验证中。在集群内重采样并没有提高性能。我们建议在人员级别而不是访问级别上划分聚类数据，以确保在预期使用中表现出色，并在模型开发时准确估计未来表现。