Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for detecting rater biases, or differential rater functioning (DRF). The purpose of this study is to illustrate and explore the sensitivity of DRF indices in realistic sparse rating designs that have been documented in the literature that include different types and levels of connectivity among raters and students. The results indicated that it is possible to detect DRF in sparse rating designs, but the sensitivity of DRF indices varies across designs. We consider the implications of our findings for practice related to monitoring raters in performance assessments.

评估者介导的评估中的实际限制限制了完整数据的可用性。相反，大多数评分程序包括对每个表现的一个或两个评分，评分者之间的表现重叠或链接多项选择项目集以促进模型估计。这些不完整的评分设计为检测评估者偏差或差异评估者功能 (DRF) 带来了挑战。本研究的目的是说明和探索 DRF 指数在现实稀疏评分设计中的敏感性，这些设计已在文献中记录，包括评分者和学生之间不同类型和级别的连接。结果表明，可以在稀疏评级设计中检测 DRF，但 DRF 指数的敏感性因设计而异。