Learners’ progress within computer-supported collaborative learning environments is typically measured via analysis and interpretation of quantitative web interaction measures. However, the usefulness of these “proxies for learning” is questioned as they do not necessarily reflect critical thinking—an essential component of collaborative learning. Research indicates that pedagogical content analysis schemes have value in measuring critical discourse in small scale, formal, online learning environments, but research using these methods on high volume, informal, Massive Open Online Course (MOOC) forums is less common. The challenge in this setting is to develop valid and reliable indicators that operate successfully at scale. In this study, we test two established coding schemes used for the pedagogical content analysis of online discussions in a large-scale review of MOOC comment data. Pedagogical Scores are derived from manual ratings applied to comments by raters and correlated with automatically derived linguistic and interaction indicators. Results show that the content analysis methods are reliable, and are very strongly correlated with each other, suggesting that their specific format is not significant in this setting. In addition, the methods are strongly associated with the relevant linguistic indicators of higher levels of learning and have weaker correlations with other linguistic and interaction metrics. This suggests promise for further research using Machine Learning techniques, with the goal of providing realistic feedback to instructors, learners, and learning designers.

中文翻译：

---

批判性思维正在发生吗？测试应用于MOOC论坛的内容分析方案

学习者在计算机支持的协作学习环境中的进步通常是通过定量网络交互措施的分析和解释来衡量的。然而，这些“学习代理”的有用性受到质疑，因为它们不一定反映批判性思维——协作学习的重要组成部分。研究表明，教学内容分析方案在衡量小规模、正式、在线学习环境中的批判性话语方面具有价值，但在大量、非正式、大规模开放在线课程 (MOOC) 论坛上使用这些方法的研究并不常见。这种情况下的挑战是开发有效和可靠的指标，并在规模上成功运行。在这项研究中，我们在大规模的 MOOC 评论数据审查中测试了两种既定的编码方案，用于在线讨论的教学内容分析。教学分数源自评分者对评论的手动评分，并与自动导出的语言和互动指标相关联。结果表明，内容分析方法是可靠的，并且相互之间具有很强的相关性，这表明它们的特定格式在这种情况下并不重要。此外，这些方法与较高学习水平的相关语言指标密切相关，与其他语言和交互指标的相关性较弱。这表明使用机器学习技术进行进一步研究的前景，目标是向教师、学习者、