Abstract In practices of direct assessment of writing ability, the variability of human decision-making during scoring poses great challenges to the validity of assessment (Kane, 2006). The variables causing differences in individual raters’ scoring interpretations have been widely investigated (e.g. Eckes, 2012; Wolfe et al., 2016). However, the issue of how raters negotiate to resolve discrepancies has not received attention although rater negotiation is a widely used score resolution method. As it has been emphasized by scholars interested in the argumentative behavior of raters (e.g. Trace et al., 2017), a systematic analysis of score negotiations will enable us to analyze the dependability of score negotiations. The purpose of this study is twofold: to present a thorough analysis of the argumentative structure of rater discrepancy resolution discussions with a view to understanding their underlying dynamics, and to investigate whether the elements of the argumentative structure of negotiations differ from research settings to authentic score resolution practices. In line with this aim, rater negotiations following a written test at the language school of an English-medium university were analyzed within the framework of Argumentation Theory by Toulmin (1958) and Walton (2005, 2016). The negotiation data were obtained from 99 recorded rater discussions among 30EFL teachers, and transcribed, coded and categorized into argumentative discussion moves. A Rater Negotiation Scheme (RNS) was developed through a recursive data analysis and categorization process, and it was validated through field-testing in authentic settings. The findings have implications both for research on rater negotiations and arguments on the reliability of the method.

中文翻译：

---

评分者协商方案：写作评分者如何解决分数差异

摘要 在直接评估写作能力的实践中，评分过程中人类决策的可变性对评估的有效性提出了巨大挑战（Kane，2006）。导致个体评分者评分解释差异的变量已得到广泛研究（例如 Eckes，2012；Wolfe 等，2016）。然而，尽管评分者协商是一种广泛使用的分数解决方法，但评分者如何协商以解决差异的问题并未受到关注。正如对评分者的争论行为感兴趣的学者所强调的那样（例如 Trace 等，2017），对评分谈判的系统分析将使我们能够分析评分谈判的可靠性。这项研究的目的有两个：对评分者差异解决讨论的论证结构进行全面分析，以了解其潜在的动态，并调查谈判的论证结构的要素是否从研究环境到真实的分数解决实践有所不同。为了实现这一目标，在 Toulmin (1958) 和 Walton (2005, 2016) 的论证理论框架内，对在一所英语语言学校的语言学校进行笔试后的评分者谈判进行了分析。谈判数据是从 30 位 EFL 教师之间的 99 次记录的评分者讨论中获得的，并转录、编码并分类为议论性讨论动作。通过递归数据分析和分类过程开发了评估者谈判方案 (RNS)，并通过真实环境中的现场测试对其进行了验证。这些发现对评估者谈判的研究和关于该方法可靠性的争论都有影响。