The use of mixed-format tests made up of multiple-choice (MC) items and constructed response (CR) items is popular in large-scale testing programs, including the National Assessment of Educational Progress (NAEP) and many district- and state-level assessments in the United States. Rater effects, or raters’ scoring tendencies that result in performances receiving different scores than are warranted given their quality, are concerns for the interpretation of scores on CR items. However, there are few published studies in which researchers have systematically considered the impact of ignoring rater effects when they are present on estimates of student ability using large-scale mixed-format assessments. Using results from an analysis of NAEP data, we systematically explored the impacts of rater effects on student achievement estimates. Our results suggest that in conditions that reflect many large-scale mixed-format assessments, directly modeling rater effects yields more accurate student achievement estimates than estimation procedures that do not incorporate raters. We consider the implications of our findings for research and practice.

中文翻译：

---

检查在混合格式测试中忽略评分者效应的影响

由多项选择 (MC) 项目和构建响应 (CR) 项目组成的混合格式测试的使用在大型测试项目中很受欢迎，包括国家教育进步评估 (NAEP) 以及许多地区和州美国的水平评估。评分者的影响，或评分者的评分倾向，导致表演获得不同的分数，而不是考虑到他们的质量，这是对 CR 项目分数的解释的关注点。然而，很少有已发表的研究表明，研究人员系统地考虑了在使用大规模混合格式评估时忽略评估者效应对学生能力估计的影响。利用对 NAEP 数据的分析结果，我们系统地探讨了评估者效应对学生成绩估计的影响。我们的结果表明，在反映许多大规模混合格式评估的条件下，直接模拟评估者效果比不包含评估者的评估程序产生更准确的学生成绩估计。我们考虑我们的发现对研究和实践的影响。