ABSTRACT

Cross-validation assesses the predictive ability of a model, allowing one to rank models accordingly. Although the nonparametric bootstrap is almost always used to assess the variability of a parameter, it can be used as the basis for cross-validation if one keeps track of which items were not selected in a given bootstrap iteration. The items which were selected constitute the training data and the omitted items constitute the testing data. This bootstrap cross-validation (BS-CV) allows model selection to be made on the basis of predictive ability by comparing the median values of ensembles of summary statistics of testing data. BS-CV is herein demonstrated using several summary statistics, including a new one termed the simple metric for prediction quality (SMPQ), and using the warfarin data included in the Monolix distribution with 13 pharmacokinetics (PK) models and 12 pharmacodynamics (PD) models. Of note the two best PK models by AIC had the worst predictive ability, underscoring the danger of using single realizations of a random variable (such as AIC) as the basis for model selection. Using these data BS-CV was able to discriminate between similar indirect response models (inhibition of input versus stimulation of output). This could be useful in situations in which the mechanism of action is unknown (unlike warfarin).

摘要

交叉验证评估模型的预测能力，允许相应地对模型进行排名。尽管非参数引导程序几乎总是用于评估参数的可变性，但如果跟踪给定引导程序迭代中未选择哪些项目，它可以用作交叉验证的基础。选择的项目构成训练数据，省略的项目构成测试数据。这种引导交叉验证 (BS-CV) 允许通过比较测试数据的汇总统计集合的中值，在预测能力的基础上进行模型选择。BS-CV 在此使用几个汇总统计数据进行演示，包括一种称为预测质量简单度量 (SMPQ) 的新统计数据，并使用包含在 Monolix 分布中的华法林数据以及 13 个药代动力学 (PK) 模型和 12 个药效学 (PD) 模型。值得注意的是，AIC 的两个最佳 PK 模型的预测能力最差，这凸显了使用随机变量（例如 AIC）的单一实现作为模型选择基础的危险。使用这些数据，BS-CV 能够区分相似的间接响应模型（输入抑制与输出刺激）。这在作用机制未知的情况下（与华法林不同）可能很有用。使用这些数据，BS-CV 能够区分相似的间接响应模型（输入抑制与输出刺激）。这在作用机制未知的情况下（与华法林不同）可能很有用。使用这些数据，BS-CV 能够区分相似的间接响应模型（输入抑制与输出刺激）。这在作用机制未知的情况下（与华法林不同）可能很有用。