ABSTRACT

With the growing use of virtual learning environments (VLE), innovative methods to evaluate their performance are increasingly needed. A key difficulty in evaluating VLE using system logs is the large heterogeneity of usage patterns. The current study demonstrates an approach to classify complex patterns of student-level and classroom-level usage with latent class analysis, then estimate average treatment effects (ATEs) of membership in student or classroom classes, as well as joint effects. The approach accounts for uncertainty of latent classes with a three-step method, and nonrandom selection into classes using inverse probability weighting. We demonstrate the approach with an analysis of usage of an Algebra VLE and estimate causal effects of latent class membership on a high-stakes Algebra I standardized assessment. Challenges of using system logs for evaluation of VLE are discussed with respect to measurement error, construct validity, latent classes enumeration, and comparison of classes with respect to distal outcomes.

摘要

随着虚拟学习环境 (VLE) 的使用越来越多，越来越需要创新的方法来评估其性能。使用系统日志评估 VLE 的一个主要困难是使用模式的巨大异质性。当前的研究展示了一种方法，通过潜在班级分析对学生级和课堂级使用的复杂模式进行分类，然后估计学生或课堂班级成员的平均治疗效果 (ATE) 以及联合效应。该方法使用三步法解决潜在类别的不确定性，并使用逆概率加权非随机选择类别。我们通过分析代数 VLE 的使用来演示该方法，并估计潜在类成员资格对高风险代数 I 标准化评估的因果影响。