Abstract

More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect estimates conducted using a CITS approach correspond to a randomized trial, varying (1) the approach to modeling the pre-intervention trend, (2) the approach used to select comparison cases, and (3) the number of years between the implementation of the intervention and measurement of the outcome of interest. Our analyses demonstrate that CITS can come close to replicating experimental findings. However, analytic decisions matter. Picking a comparison group that accounts for local context, past performance, and school demographics can reduce bias and reliance on model selection. Likewise, fitting nonlinear trend models can lead to substantial bias in short time series analyses. Finally, the performance of CITS declines for outcomes measured further after the introduction of the intervention.

摘要

可获得比以往更多的学校绩效数据，这为有兴趣通过实施比较间断时间序列（CITS）设计以较低成本快速评估学校级干预措施有效性的应用研究人员提供了新的可能性。我们研究了使用CITS方法进行的效果评估在多大程度上与随机试验相对应，改变了（1）干预前趋势建模的方法，（2）选择比较案例的方法以及（3）数量从实施干预措施到测量感兴趣的结果之间的年数。我们的分析表明，CITS可以接近复制实验结果。但是，分析决策很重要。选择一个能够说明当地情况，过往表现和学校人口统计信息的比较组，可以减少偏见和对模型选择的依赖。同样，拟合非线性趋势模型可能会在短时间序列分析中导致较大的偏差。最后，对于采用干预措施后进一步衡量的结果，CITS的绩效下降。