External validity is often questionable in empirical research, especially in randomized experiments due to the trade-off between internal validity and external validity. To quantify the robustness of external validity, one must first conceptualize the gap between a sample that is fully representative of the target population (i.e., the ideal sample) and the observed sample. Drawing on Frank & Min (2007) and Frank et al. (2013), I define such gap as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST) in this study. The probability of invalidating a causal inference due to limited external validity, i.e., the PEV, is the probability of failing to reject the null hypothesis based on the ideal sample provided the null hypothesis has been rejected based on the observed sample. This study illustrates the guideline and the procedure of evaluating external validity with the PEV through an empirical example (i.e., Borman et al. (2008)). Specifically, one would be able to locate the threshold of the unobserved sample statistic that would make the PEV higher than a desired value and use this information to characterize the unobserved sample that would render external validity of the research in question less robust. The PEV is shown to be linked to statistical power when the NHST is thought to be based on the ideal sample.

中文翻译：

---

关于由于有限的外部有效性而使因果推论无效的概率

外部有效性在实证研究中经常受到质疑，特别是在随机实验中，由于内部有效性和外部有效性之间的权衡。要量化外部效度的稳健性，首先必须概念化完全代表目标人群的样本（即理想样本）与观察样本之间的差距。借鉴 Frank & Min (2007) 和 Frank 等人。(2013)，我将这种差距定义为未观察到的样本，并打算在本研究中量化其与零假设统计检验 (NHST) 的关系。由于有限的外部有效性（即 PEV）而使因果推论无效的概率是基于理想样本无法拒绝原假设的概率，前提是已根据观察到的样本拒绝原假设。本研究通过一个实证例子（即，Borman 等人（2008 年））说明了使用 PEV 评估外部效度的指南和程序。具体来说，人们将能够定位未观察到的样本统计量的阈值，该阈值将使 PEV 高于所需值，并使用此信息来表征未观察到的样本，这将使相关研究的外部有效性变得不那么稳健。当 NHST 被认为是基于理想样本时，PEV 与统计功效相关联。人们将能够定位未观察到的样本统计量的阈值，该阈值会使 PEV 高于所需值，并使用此信息来表征未观察到的样本，这将使相关研究的外部有效性变得不那么稳健。当 NHST 被认为是基于理想样本时，PEV 与统计功效相关联。人们将能够定位未观察到的样本统计量的阈值，该阈值会使 PEV 高于所需值，并使用此信息来表征未观察到的样本，这将使相关研究的外部有效性变得不那么稳健。当 NHST 被认为是基于理想样本时，PEV 与统计功效相关联。