ABSTRACT

Formative assessments are an important component of massive open online courses (MOOCs), online courses with open access and unlimited student participation. Accurate conclusions on students’ proficiency via formative, however, face several challenges: (a) students are typically allowed to make several attempts; and (b) student performance might be affected by other variables, such as interest. Thus, neglecting the effects of attempts and interest in proficiency evaluation might result in biased conclusions. In this study, we try to solve this limitation and propose two extensions of the common psychometric model, the Rasch model, by including the effects of attempts and interest. We illustrate these extensions using real MOOC data and evaluate them using cross-validation. We found that (a) the effects of attempts and interest on the performance are positive on average but both vary among students; (b) a part of the variance in proficiency parameters is due to variation between students in the effect of interest; and (c) the overall accuracy of prediction of student’s item responses using the extensions is 4.3% higher than using the Rasch model.

摘要

形成性评估是大规模开放在线课程（MOOC），具有开放访问权限和无限学生参与的在线课程的重要组成部分。然而，通过形成方式准确地得出学生的能力结论面临着几个挑战：（a）通常允许学生进行多次尝试；（b）学生的表现可能会受到其他变量（例如兴趣）的影响。因此，忽略尝试和兴趣对能力评估的影响可能会导致结论有偏差。在这项研究中，我们试图解决这一局限性，并通过包括尝试和兴趣的影响，提出了常见心理测量模型Rasch模型的两个扩展。我们使用真实的MOOC数据说明这些扩展，并使用交叉验证对其进行评估。我们发现：（a）尝试和兴趣对表现的影响平均是积极的，但在学生之间两者都不同；（b）能力参数的部分差异是由于学生之间在兴趣效果方面的差异所致；（c）使用扩展名预测学生的项目反应的总体准确性比使用Rasch模型高4.3％。