Differences in rater judgments that are systematically related to construct-irrelevant characteristics threaten the fairness of rater-mediated writing assessments. Accordingly, it is essential that researchers and practitioners examine the degree to which the psychometric quality of rater judgments is comparable across test-taker subgroups. Nonparametric procedures for exploring these differences are promising because they allow researchers and practitioners to examine important characteristics of ratings without potentially inappropriate parametric transformations or assumptions. This study illustrates a nonparametric method based on Mokken scale analysis (MSA) that researchers and practitioners can use to identify and explore differences in the quality of rater judgments between subgroups of test-takers. Overall, the results suggest that MSA provides insight into differences in rating quality across test-taker subgroups based on demographic characteristics. Differences in the degree to which raters adhere to basic measurement properties suggest that the interpretation of ratings may vary across subgroups. The implications of this study for research and practice are discussed.

中文翻译：

---

在评分者介导的写作评估中探索不同考生子组评分质量差异的非参数程序

与结构无关特征系统相关的评分者判断的差异威胁到评分者介导的写作评估的公平性。因此，研究人员和从业者必须检查评估者判断的心理测量质量在考生子组中的可比程度。探索这些差异的非参数程序很有前景，因为它们允许研究人员和从业人员检查评级的重要特征，而无需进行潜在的不适当的参数转换或假设。本研究说明了一种基于 Mokken 量表分析 (MSA) 的非参数方法，研究人员和从业人员可以使用该方法来识别和探索不同考生子组之间评分者判断质量的差异。全面的，结果表明，MSA 提供了基于人口统计特征的测试者子组之间评分质量差异的洞察力。评分者坚持基本测量属性的程度的差异表明，对评分的解释可能因亚组而异。讨论了这项研究对研究和实践的影响。