Abstract

This article assesses the likely generalizability of educational treatment-effect estimates from regression discontinuity designs (RDDs) when treatment assignment is based on academic pretest scores. Our assessment uses data on outcome and pretest measures from six educational experiments, ranging from preschool through high school, to estimate RDD generalization bias. We then compare those estimates (reported as standardized effect sizes) with the What Works Clearinghouse (WWC) standard for acceptable bias size (≤ 0.05$\sigma )$ for two target populations, one spanning a half–standard deviation pretest-score range and another spanning a full–standard deviation pretest-score range. Our results meet this standard for all 18 study/outcome/pretest scenarios examined given the narrower target population, and for 15 scenarios given the broader target population. Fortunately, two of the three exceptions represent pronounced “ceiling effects” that can be identified empirically, making it possible to avoid unwarranted RDD generalizations, and the third exception is very close to the WWC standard.

摘要

当治疗分配基于学术预测成绩时，本文评估了回归不连续性设计（RDD）对教育治疗效果评估的可能概括性。我们的评估使用从学前班到高中的六个教育实验中有关结果和预测措施的数据来估计RDD普遍偏见。然后，我们将这些估算值（报告为标准化效应量）与What Works信息交换所（WWC）标准的可接受偏差量（≤0.05）进行比较$\sigma ）$对于两个目标人群，一个跨越标准偏差前半数测试范围，另一个跨越标准偏差前半数测试范围。在目标人群较窄的情况下，对于所研究的所有18种研究/结果/预测试方案，我们的结果均符合此标准。幸运的是，三个例外中的两个代表明显的“天花板效应”，可以根据经验进行识别，从而可以避免不必要的RDD泛化，并且第三个例外与WWC标准非常接近。