ABSTRACT

In education, multisite trials involve randomization of pupils into intervention and comparison groups within schools. Most analytical models in multisite educational trials ignore that the impact of an intervention may be school dependent. This study investigates the impact of statistical models on the uncertainty associated with an effect size using comparable outcomes and covariates from ten multisite educational trials funded by the UK’s Education Endowment Foundation. Ordinary least squares (OLS) models often assume that the pupil’s outcomes within schools are independent, which is not always true. Multilevel models address this limitation by incorporating heterogeneity between schools to account for intra-school dependency. This inflates the confidence interval of an effect size obtained from the multilevel models than from an OLS model. For a multisite trial, the heterogeneity between schools also includes the differences in the expected impact of intervention between schools. Ignoring this additional school-by-intervention variation in a multisite trial could affect both its interpretation and conclusions. A robust approach to estimate the confidence intervals for effect size from multisite trials is by treating effect size as a parameter with its distribution. This paper is important for evaluating evidence from multisite trials by accounting for all sources of variability.

摘要

在教育方面，多地点试验涉及将学生随机分配到学校内的干预组和对照组中。多地点教育试验中的大多数分析模型都忽略了干预的影响可能取决于学校。本研究使用英国教育捐赠基金会资助的十个多地点教育试验的可比较结果和协变量，调查统计模型对与效应大小相关的不确定性的影响。普通最小二乘 (OLS) 模型通常假设学生在学校的成绩是独立的，但这并不总是正确的。多级模型通过结合学校之间的异质性来解决校内依赖问题，从而解决了这一限制。这使得从多级模型获得的效应大小的置信区间比从 OLS 模型获得的更大。对于多中心试验，学校之间的异质性还包括学校之间干预的预期影响的差异。在多中心试验中忽略这种额外的学校干预变化可能会影响其解释和结论。一种可靠的方法来估计来自多中心试验的效应大小的置信区间是通过将效应大小视为具有其分布的参数。本文通过考虑所有可变性来源，对于评估来自多中心试验的证据非常重要。一种可靠的方法来估计来自多中心试验的效应大小的置信区间是通过将效应大小视为具有其分布的参数。本文通过考虑所有可变性来源，对于评估来自多中心试验的证据非常重要。一种可靠的方法来估计来自多中心试验的效应大小的置信区间是通过将效应大小视为具有其分布的参数。本文通过考虑所有可变性来源，对于评估来自多中心试验的证据非常重要。