ABSTRACT

When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters’ scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters’ scoring processes, an alternative scoring method was used. With this method, rubric developers’ intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed.

摘要

在对绩效评估进行评分时，如果评分标准的应用与评分标准的预期应用不一致，则评估者可以为同一绩效分配不同的分数。给定的绩效评估分数解释假定评估者按照开发者的意图应用了评估标准，评估者的评分过程与预期的评估过程之间的不一致可能导致从这些评分中得出无效的推论。为了使评分者的评分过程标准化，使用了一种替代评分方法。使用这种方法，通过要求评估者对一系列类似于决策树的选定响应语句做出响应，可以明确明确项目开发人员的预期评分过程。为了确定评估者是否使用传统的评分标准和替代评分方法对论文进行了预期的评分，指定了具有树状结构（IRTree）的IRT模型来描述预期的评分过程，并适合每种评分方法的数据。结果表明，使用替代方法的评估者可能能够更好地按预期进行评估，因此替代方法可能是传统专栏评分的可行替代方法。讨论了IRTree模型的含义。