Abstract This study adopts a novel approach to investigate perceptions of assessment criteria relevance in differentiating writing performance levels. Experienced writing assessors were asked to directly compare pairs of performances. For each comparison, assessors were asked to determine which performance was better and to record which aspects of writing were used to make determinations. To do so, assessors referred to a marking rubric containing 10 criteria. By triangulating direct/explicit qualitative comparisons of performances against indirect/implied comparisons using a rubric, the study aims to provide unique insights into the perceived relevance of criteria. Whereas the rubric presupposes relatively equal relevance of all criteria, in direct comparisons participating assessors perceive certain criteria relevant more often than others. In particular, conventions of writing, such as punctuation and spelling, were perceived relevant less often than authorial aspects. In combination with descriptions of qualitative gradations contained in the rubric itself, the information obtained from the study aims to assist with identifying and delineating criteria perceived relevant to developmental levels by the expert assessors. These and other findings have potential implications for selecting which criteria to use in narrative writing rubrics and they suggest advantages of granting assessors discretion in their selection and application of criteria.

中文翻译：

---

评估者对评估标准相关性的看法

摘要 本研究采用一种新颖的方法来调查对区分写作表现水平的评估标准相关性的看法。经验丰富的写作评估员被要求直接比较两对表演。对于每次比较，评估员都被要求确定哪种表现更好，并记录使用写作的哪些方面来做出决定。为此，评估员参考了一个包含 10 个标准的评分标准。通过使用量规对绩效的直接/显式定性比较与间接/隐含比较进行三角测量，该研究旨在提供对标准感知相关性的独特见解。尽管该评分准则以所有标准的相关性相对相等为前提，但在直接比较中，参与的评估者比其他标准更频繁地认为某些标准相关。特别是，与作者方面相比，标点符号和拼写等写作惯例被认为更不相关。结合评分量规本身包含的定性等级描述，从研究中获得的信息旨在帮助识别和描述专家评估员认为与发展水平相关的标准。这些和其他发现对于选择在叙事写作准则中使用哪些标准具有潜在影响，并且它们表明授予评估者在选择和应用标准时自由裁量权的优势。从研究中获得的信息旨在帮助确定和描述专家评估员认为与发展水平相关的标准。这些和其他发现对于选择在叙事写作准则中使用哪些标准具有潜在影响，并且它们表明授予评估者在选择和应用标准时自由裁量权的优势。从研究中获得的信息旨在帮助确定和描述专家评估员认为与发展水平相关的标准。这些和其他发现对于选择在叙事写作准则中使用哪些标准具有潜在影响，并且它们表明授予评估者在选择和应用标准时自由裁量权的优势。