Abstract

Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same “potential” usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.

摘要

对教育技术的随机评估将日志数据作为副产品产生：有关学生和老师使用情况的高度精细的数据。这些数据集可以阐明因果机制，影响异质性或最佳使用。但是，存在方法上的挑战：实现不是随机的，仅针对治疗组进行定义，日志数据集具有复杂的结构。本文讨论了三种方法来帮助克服这些问题。一种方法是使用治疗组的数据来评估观察性研究中使用对结局的影响。另一个因果中介分析估计了使用在推动整体效果中的作用。最后，校长分层估计了具有相同“潜在”用法的学生群体的总体效果。我们使用以下三种方法，通过对认知导师代数I课程评估的提示数据进行分析，结果可能会相互矛盾：观察性研究和调解分析表明提示降低了测验分数，而主要分层发现治疗效果可能与更高的发生率相关提示请求。我们讨论了这些混杂的结论，并提供了更广泛的方法学建议。