Abstract

Rubrics have been suggested as a means to foster students’ evaluative judgement, the capacity to appraise their own work and that of others; however, empirical evidence of rubrics’ effectiveness is still emerging. This paper contributes findings from a randomised controlled experiment on the effect of rubrics on evaluative judgement. Participants were randomly assigned to one of two groups: a control group which evaluated peer-authored learning resources without the use of a rubric and an experiment group which carried out the evaluation using a three-item rubric based on (1) alignment with course content, (2) accuracy and (3) clarity. Both groups were asked to rate their confidence and provide comments to justify their scoring. The results showed a small effect size in increasing average agreement on the quality of learning resources in the experiment group. Analysis of comments reveals that criteria in and beyond the rubric guided participants’ ratings of quality. The study provides evidence of the impact of rubrics on students’ evaluative judgement and an example of how data-driven approaches and learning analytics can inform actionable design choices for embedding pedagogically supported strategies derived from the literature into actively operating educational technologies.

摘要

已建议将量规作为培养学生评价判断、评价自己和他人工作的能力的一种手段；然而，关于标准有效性的经验证据仍在不断涌现。本文提供了一项关于量规对评价性判断影响的随机对照实验的结果。参与者被随机分配到两组中的一组：对照组，在不使用评分量规的情况下评估同行创作的学习资源；实验组使用基于（1）与课程内容一致的三项评分量规进行评估，（2）准确性和（3）清晰度。两组都被要求评估他们的信心并提供评论以证明他们的得分。结果表明，在增加实验组学习资源质量的平均一致性方面，效应量很小。对评论的分析表明，该标准内外的标准指导了参与者对质量的评级。该研究提供了量规对学生评估判断的影响的证据，以及数据驱动方法和学习分析如何为可操作的设计选择提供信息，以便将源自文献的教学支持策略嵌入到积极运作的教育技术中。