Abstract Peer assessment activities might be one of the few personalized assessment alternatives to the implementation of auto-graded activities at scale in Massive Open Online Course (MOOC) environments. However, teacher's motivation to implement peer assessment activities in their courses might go beyond the most straightforward goal (i.e., assessment), as peer assessment activities also have other side benefits, such as showing evidence and enhancing the critical thinking, comprehension or writing capabilities of students. However, one of the main drawbacks of implementing peer review activities, especially when the scoring is meant to be used as part of the summative assessment, is that it adds a high degree of uncertainty to the grades. Motivated by this issue, this paper analyses the reliability of all the peer assessment activities performed as part of the MOOC platform of the Spanish University for Distance Education (UNED) UNED-COMA. The following study has analyzed 63 peer assessment activities from the different courses in the platform, and includes a total of 27,745 validated tasks and 93,334 peer reviews. Based on the Krippendorff's alpha statistic, which measures the agreement reached between the reviewers, the results obtained clearly point out the low reliability, and therefore, the low validity of this dataset of peer reviews. We did not find that factors such as the topic of the course, number of raters or number of criteria to be evaluated had a significant effect on reliability. We compare our results with other studies, discuss about the potential implications of this low reliability for summative assessment, and provide some recommendations to maximize the benefit of implementing peer activities in online courses.

中文翻译：

---

在 MOOC 平台中审查和分析同行评审间的可靠性

摘要 同行评估活动可能是在大规模开放在线课程 (MOOC) 环境中大规模实施自动评分活动的少数个性化评估替代方案之一。然而，教师在课程中实施同伴评估活动的动机可能超出最直接的目标（即评估），因为同伴评估活动还有其他好处，例如展示证据和提高批判性思维、理解或写作能力。学生们。然而，实施同行评审活动的主要缺点之一，尤其是当评分旨在用作总结性评估的一部分时，它会给成绩增加高度的不确定性。受这个问题的启发，本文分析了作为西班牙远程教育大学 (UNED) UNED-COMA 的 MOOC 平台的一部分进行的所有同行评估活动的可靠性。以下研究分析了平台中不同课程的 63 项同行评估活动，包括总共 27,745 项经过验证的任务和 93,334 项同行评议。基于 Krippendorff 的 alpha 统计量，它衡量审稿人之间达成的协议，获得的结果清楚地指出了低可靠性，因此，同行评议数据集的有效性低。我们没有发现诸如课程主题、评分者数量或要评估的标准数量等因素对可靠性有显着影响。我们将我们的结果与其他研究进行比较，