ABSTRACT

Scoring procedures for many rater-mediated performance assessments include score resolution procedures in which a third rater adjudicates discrepancies between two raters’ ratings of the same performance. There are numerous approaches for calculating resolved scores that involve different combinations of the original and third ratings. Using data from two content-area writing assessments, we examined the impact of applying no resolution and applying three resolution techniques: TQ, Parity, and the Operational resolution method used in the assessment program on the values and psychometric quality of student achievement estimates. We observed substantial differences in person fit, a model-based measure of the psychometric quality of student achievement estimates, across methods. Our results highlight the importance of examining person fit as part of evaluating the psychometric quality of rater-mediated performance assessments in which score resolution procedures are employed.

摘要

许多评估者介导的绩效评估的计分程序包括分数解决程序，其中第三评估者裁定两个评估者对同一绩效的评估之间的差异。有许多计算解决分数的方法，这些方法涉及原始评级和第三评级的不同组合。使用来自两个内容区域写作评估的数据，我们研究了不采用解决方案和应用三种解决方案技术（TQ，奇偶校验和评估方案中使用的操作解决方案方法）对学生成绩评估值和心理计量质量的影响。我们观察到不同方法在人的适应度方面存在重大差异，这是一种基于模型的学生成绩估算的心理计量质量度量。