How best to estimate the accuracy of a predictive rule has been a longstanding question in statistics. Approaches to this task range from simple methods like Mallow’s Cp to algorithmic techniques like cross-validation; see Arlot and Celisse (2010), Efron (1983, 2004), Hastie, Tibshirani, and Friedman (2009), Mallows (1973), and references therein. Rosset and Tibshirani (2019) contribute to this discussion by considering how some classical results on the “optimism” of the apparent error of a predictive rule, that is, the amount by which the training set error of a fitted statistical predictor is expected to underestimate its test set error, change when we consider a randomversus fixed-X sampling design. This is a welcome addition to the literature as, in modern statistical settings, we often need to work with large observational datasets that were incidentally collected as a by-product of some other task, and in these cases random-X modeling is more appropriate than the classical fixed-X approach. There are two reasons a statistician may want to estimate the accuracy of a predictive model. One is to simply understand the quality of its predictions: For example, a company may need to choose whether to purchase a new forecasting tool, and want to evaluate its accuracy to better understand the value of the tool for its business. Another motivation is model selection: Crossvalidation and related methods are often used to choose between competing predictive rules, or to set the complexity parameter with methods like the lasso (Hastie, Tibshirani, and Friedman 2009; Chetverikov, Liao, and Chernozhukov 2016). For the first task, we in fact need to accurately estimate the accuracy of the predictive rule itself, and the results of Rosset and Tibshirani (2019) are focused on this task. For the second, however, we only need to compare the accuracy of two competing rules; and this statistical task ends up having fairly different properties than risk estimation. In this note, I compare properties of K-fold cross-validation for both risk estimation and model comparison under randomX asymptotics. We have access to independent and identically distributed samples (Xi, Yi) ∈ X × R, and want to predict Yi from Xi under squared error loss. The optimal predictive rule is the conditional response function μ(x) = E [Y ∣∣ X = x]. For simplicity, I will focus on evaluating models μ̂(x) whose rootmean-squared excess error E[(μ̂(X) − μ(X))2]1/2 decays with sample size n as n−γ , for some exponent 1/4 < γ < 1/2. In

中文翻译：

---

交叉验证、风险估计和模型选择：Rosset 和 Tibshirani 对论文的评论

如何最好地估计预测规则的准确性一直是统计学中的一个长期问题。此任务的方法范围从简单的方法（如 Mallow's Cp）到算法技术（如交叉验证）；参见 Arlot 和 Celisse (2010)、Efron (1983、2004)、Hastie、Tibshirani 和 Friedman (2009)、Mallows (1973) 以及其中的参考资料。Rosset 和 Tibshirani（2019 年）通过考虑一些关于预测规则表观误差“乐观”的经典结果，即拟合统计预测器的训练集误差预计低估的量，对这一讨论做出了贡献当我们考虑随机与固定 X 抽样设计时，它的测试集误差会发生变化。这是对文献的一个受欢迎的补充，因为在现代统计环境中，我们经常需要使用大型观测数据集，这些数据集是作为其他任务的副产品偶然收集的，在这些情况下，随机 X 建模比经典的固定 X 方法更合适。统计学家可能想要估计预测模型的准确性有两个原因。一种是简单地了解其预测的质量：例如，一家公司可能需要选择是否购买新的预测工具，并希望评估其准确性以更好地了解该工具对其业务的价值。另一个动机是模型选择：交叉验证和相关方法通常用于在相互竞争的预测规则之间进行选择，或使用套索等方法设置复杂性参数（Hastie、Tibshirani 和 Friedman 2009；Chetverikov、Liao 和 Chernozhukov 2016）。对于第一个任务，我们实际上需要准确估计预测规则本身的准确度，Rosset 和 Tibshirani (2019) 的结果集中在这个任务上。然而，对于第二个，我们只需要比较两个竞争规则的准确性；并且这个统计任务最终具有与风险估计完全不同的特性。在这篇笔记中，我比较了在 randomX 渐近下风险估计和模型比较的 K 折交叉验证的特性。我们可以访问独立同分布的样本 (Xi, Yi) ∈ X × R，并希望在平方误差损失下从 Xi 预测 Yi。最优预测规则是条件响应函数 μ(x) = E [Y ∣∣ X = x]。为简单起见，我将重点评估模型 μ̂(x)，其均方根超差 E[(μ̂(X) − μ(X))2]1/2 随样本大小 n 衰减为 n−γ ，对于某些指数 1/4 < γ < 1/2。在