Abstract Analytic rubrics for writing assessments are intended to provide diagnostic information regarding students’ strengths and weaknesses related to several domains, such as the meaning and mechanics of their composition. Although individual domains refer to unique aspects of student writing, the same rating scales are often applied across different domains. Accordingly, the interpretation of rating scale categories is intended to be invariant across domains. However, this hypothesis is rarely examined empirically. The purpose of this study is to illustrate and explore methods for evaluating the degree to which raters apply a common rating scale consistently across domains in analytic writing assessments. Data from a rater training procedure for a rater-mediated writing assessment serve as a case study for the illustrative analyses. Results indicated a lack of invariance in rating scale category functioning across domains for several raters in both examples. Implications are discussed in terms of how evidence of differential rating scale category use can be used to inform rater training and assessment development procedures.

中文翻译：

---

评分者在写作评估中是否一致地使用跨分析量规领域的评分量表类别？

摘要 写作评估的分析量规旨在提供有关学生在多个领域的优势和劣势的诊断信息，例如其作文的含义和机制。虽然个别领域指的是学生写作的独特方面，但相同的评分量表通常适用于不同的领域。因此，评级量表类别的解释旨在跨域保持不变。然而，这个假设很少被实证检验。本研究的目的是说明和探索评估评分者在分析性写作评估中跨领域一致应用通用评分量表的程度的方法。来自评估者中介写作评估的评估者培训程序的数据可作为说明性分析的案例研究。结果表明，在两个示例中，对于多个评分者而言，跨域的评分量表类别缺乏不变性。在如何使用差异评分量表类别的证据为评分者培训和评估开发程序提供信息方面讨论了影响。