Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

模型校准对于临床预测模型的成功和安全至关重要，但会随着时间的推移而恶化，以响应临床环境的动态特性。为了支持知情的、数据驱动的模型更新策略，我们提出并评估了一个校准漂移检测系统。开发了使用优化的在线随机梯度下降来维护动态校准曲线以及使用自适应滑动窗口检测增加的误校准的方法。这些方法可推广以支持使用各种学习算法开发的各种预测模型，并可定制以满足临床用例的独特需求。在模拟和案例研究中，我们的系统准确地检测到校准漂移。当检测到漂移时，我们的系统通过在可能适合模型更新的最近数据窗口中包含信息，进一步提供可操作的警报。模拟表明，这些窗口主要由漂移开始后累积的数据组成，支持窗口用于模型更新的潜在效用。通过在校准恶化而不是按预定时间表进行模型更新，我们的漂移检测系统的实施可以最大限度地减少模型准确性不足的过渡期，并将分析资源集中在最需要关注的那些模型上。