First, we analyze the variance of the Cross Validation (CV)-based estimators used for estimating the performance of classification rules. Second, we propose a novel estimator to estimate this variance using the Influence Function (IF) approach that had been used previously very successfully to estimate the variance of the bootstrap-based estimators. The motivation for this research is that, as the best of our knowledge, the literature lacks a rigorous method for estimating the variance of the CV-based estimators. What is available is a set of ad-hoc procedures that have no mathematical foundation since they ignore the covariance structure among dependent random variables. The conducted experiments show that the IF proposed method has small RMS error with some bias. However, surprisingly, the ad-hoc methods still work better than the IF-based method. Unfortunately, this is due to the lack of enough smoothness if compared to the bootstrap estimator. This opens the research for three points: (1) more comprehensive simulation study to clarify when the IF method wins or looses; (2) more mathematical analysis to figure out why the ad-hoc methods work well; and (3) more mathematical treatment to figure out the connection between the appropriate amount of “smoothness” and decreasing the bias of the IF method.

首先，我们分析用于估计分类规则性能的基于交叉验证（CV）的估计量的方差。其次，我们提出了一种新颖的估计器，使用影响函数（IF）方法估计此方差，该方法以前非常成功地用于估计基于自举的估计器的方差。进行这项研究的动机是，据我们所知，文献缺乏一种精确的方法来估计基于CV的估计器的方差。可用的是一组没有数学基础的即席程序，因为它们忽略了相关随机变量之间的协方差结构。进行的实验表明，提出的中频方法具有较小的均方根误差，并具有一定的偏差。但是，令人惊讶的是，临时方法仍然比基于IF的方法更好。不幸的是，这是由于与自举估计器相比缺乏足够的平滑度。这为研究开辟了三点：（1）更全面的仿真研究，以明确IF方法何时获胜或失败；以及 （2）进行更多的数学分析，以找出临时方法为何行之有效；（3）进行更多的数学处理，以找出适当数量的“平滑度”与降低IF方法的偏差之间的联系。