ABSTRACT

To score an essay response, raters draw on previously trained skills and knowledge about the underlying rubric and score criterion. Cognitive processes such as remembering, forgetting, and skill decay likely influence rater performance. To investigate how forgetting influences scoring, we evaluated raters’ scoring accuracy on TOEFL and GRE essays. We used binomial linear mixed effect models to evaluate how the effect of various predictors such as time spent scoring each response and days between scoring sessions relate to scoring accuracy. Results suggest that for both nonoperational (i.e., calibration samples completed prior to a scoring session) and operational scoring (i.e., validity samples interspersed among actual student responses), the number of days in a scoring gap negatively affects performance. The findings, as well as other results from the models are discussed in the context of cognitive influences on knowledge and skill retention.

摘要

为了对一篇论文的回答进行评分，评分者会利用先前受过训练的技能和有关基本专栏和评分标准的知识。诸如记住，忘记和技能衰退之类的认知过程可能会影响评分者的表现。为了研究忘记因素如何影响评分，我们评估了托福和GRE论文中评分者的评分准确性。我们使用了二项式线性混合效应模型来评估各种预测变量的效果，例如花费在对每个响应评分的时间以及评分会话之间的天数与评分准确性之间的关系。结果表明，对于非操作性（即，在评分会话之前完成的校准样本）和操作性评分（即，有效性样本散布在实际学生的回答中）两者之间，评分差距中的天数会对成绩产生负面影响。调查结果，