This study examines the decision-making behaviors of raters with varying levels of experience while assessing EFL essays of distinct qualities. The data were collected from 28 raters with varying levels of rating experience and working at the English language departments of different universities in Turkey. Using a 10-point analytic rubric, each rater voice-recorded their thoughts through think-aloud protocols (TAPs) while scoring 16 essays of distinct text qualities and provided brief score explanations. Data collected from TAPs were analyzed by using a coding scheme adapted from Cumming, Kantor, and Powers (2002). The results revealed that text quality has a larger effect than rating experience on raters’ decision-making behaviors. In addition, raters prioritized aspects of style, grammar, and mechanics when rating low-quality essays, but emphasized rhetoric and their general impressions of the text for high-quality essays. Furthermore, low-experienced raters differed more in their behaviors while assessing scripts of distinct qualities than did the medium- and high-experienced groups. The findings suggest that raters’ scoring behaviors might evolve with practice, resulting in less variation in their decisions. As such, this research provides implications for developing strategy-based rater training programs, which might help to increase consistency across raters of different experience levels.

中文翻译：

---

经验和文本质量对评分者的决策行为是否重要？

本研究考察了具有不同经验水平的评分者的决策行为，同时评估了不同质量的 EFL 论文。数据是从 28 位具有不同评级经验并在土耳其不同大学英语系工作的评级员收集的。使用 10 分分析量规，每位评分者通过大声思考协议 (TAP) 记录他们的想法，同时对 16 篇具有不同文本质量的文章进行评分，并提供简短的评分解释。使用改编自 Cumming、Kantor 和 Powers (2002) 的编码方案分析从 TAP 收集的数据。结果表明，文本质量对评分者决策行为的影响大于评分经验。此外，评分者在评分低质量论文时会优先考虑风格、语法和机制等方面，而是强调修辞和他们对文本的总体印象，以便撰写高质量的文章。此外，与中等和经验丰富的群体相比，经验不足的评分者在评估不同质量的脚本时的行为差异更大。研究结果表明，评分者的评分行为可能会随着实践而发展，从而导致他们的决策变化较小。因此，这项研究为制定基于策略的评估者培训计划提供了启示，这可能有助于提高不同经验水平的评估者之间的一致性。研究结果表明，评分者的评分行为可能会随着实践而发展，从而导致他们的决策变化较小。因此，这项研究为制定基于策略的评估者培训计划提供了启示，这可能有助于提高不同经验水平的评估者之间的一致性。研究结果表明，评分者的评分行为可能会随着实践而发展，从而导致他们的决策变化较小。因此，这项研究为制定基于策略的评估者培训计划提供了启示，这可能有助于提高不同经验水平的评估者之间的一致性。