On the second day of a 2019 high-stakes English Language Arts assessment, Massachusetts 10th graders faced an essay question that was based on a passage from the novel “The Underground Railroad” and publicly characterized as racially insensitive. Though the state excluded the essay responses from student scores, an unresolved public controversy focused on whether this question created a racial bias in performance on the remaining test items. We present the results from an independent, preregistered study of this question. Our confirmatory results indicate that exposure to the controversial question is associated with a small reduction in the comparative performance of Black students on the overall test (approximately .006σ). However, we also find a wide dispersion of such effects when examining similarly small sets of test items from prior state assessments that lacked a controversial question, which suggests the 2019 assessment was not distinctive. Our approach offers a potential template that may be useful in other contexts where testing controversies occur and underscores the importance of carefully screening test items to avoid such occurrences.

中文翻译：

---

评估测试问题的影响：来自“地下铁路”争议的证据

在 2019 年高风险英语语言艺术评估的第二天，马萨诸塞州 10 年级学生面临一个作文题，题目是根据小说“地下铁路”中的一段话，公开描述为对种族不敏感。尽管该州从学生成绩中排除了论文回答，但一个悬而未决的公众争议集中在这个问题是否在其余测试项目的表现中造成了种族偏见。我们展示了对这个问题进行的一项独立的、预先注册的研究的结果。我们的验证结果表明，暴露于有争议的问题与黑人学生在整体测试中的比较表现略有下降（约 0.006σ）有关。然而，缺乏一个有争议的问题，这表明 2019 年的评估并不独特。我们的方法提供了一个潜在的模板，在发生测试争议的其他情况下可能有用，并强调了仔细筛选测试项目以避免此类事件发生的重要性。