ABSTRACT

Rater training is essential in ensuring the quality of constructed response scoring. Most of the current knowledge about rater training comes from experimental contexts with an emphasis on short-term effects. Few sources are available for empirical evidence on whether and how raters become more accurate as they gain scoring experiences or what long-term effects training can have. In this study, we addressed this research gap by tracking how the accuracies of new raters change through experience and by examining the impact of an additional training session on their accuracies in scoring calibration and monitoring essays. We found that, on average, raters’ accuracy improved with scoring experience and that individual raters differed in their accuracy trajectories. The estimated average effect of the training was an approximately six percent increase in the calibration essay accuracy. On the other hand, we observed a smaller impact on the monitoring essay accuracy. Our follow-up analysis showed that this differential impact of the additional training on the calibration and monitoring essay accuracy could be accounted for by successful gatekeeping through calibration.

摘要

评分员培训对于确保构建的反应评分质量至关重要。当前有关评估者培训的大多数知识来自实验环境，重点是短期效应。关于评估者在获得评分经验时或如何变得更准确或培训可以产生什么样的长期影响的经验证据，很少有可用的证据。在这项研究中，我们通过跟踪新评分者的准确性如何通过经验变化以及通过检查其他培训课程对他们对评分和监测论文评分的准确性的影响，来解决这一研究空白。我们发现，平均而言，评分者的准确性随着得分经验的提高而提高，并且各个评分者的准确性轨迹有所不同。估计的培训平均效果是使标准论文准确性提高了大约6％。另一方面，我们观察到对监控论文准确性的影响较小。我们的后续分析表明，额外培训对校准和监控论文准确性的不同影响可以通过通过校准成功完成关门来解决。