Using trace data to enhance Students' self-regulation: A learning analytics perspective

https://doi.org/10.1016/j.iheduc.2022.100855Get rights and content

Highlights

  • Students' digital trace data from LMS in an online learning environment can reflect self-reported SRL data in some degree.

  • Students' digital trace data is more powerful in predicting students' academic performance than self-reported SRL data.

  • Important students' learning behavior variables are identified in an online learning environment.

  • Online students' learning behavior patterns are analyzed using cluster analysis.

  • Explanations for differences between students' self-reported SRL data and the digital trace data are explored and discussed.

Abstract

The purpose of this study was to investigate whether students' self-reported SRL align with their digital trace data collected from the learning management system. This study took place in an upper-level college agriculture course delivered in an asynchronous online format. By comparing online students' digital trace data with their self-reported data, this study found that digital trace data from LMS could predict students' performance more accurately than self-reported SRL data. Through cluster analysis, students were classified into three levels based on their self-regulatory ability and the characteristics of each group were analyzed. By incorporating qualitative data, we explored possible explanations for the differences between students' self-reported SRL data and the digital trace data. This study challenges us to question the validity of existing self-reported SRL instruments. The three-cluster division of students' learning behaviors provides practical implications for online teaching and learning.

Introduction

Online education has been growing tremendously in the past decade (Van Rooij & Zirkle, 2016), and it has been playing a dominant role in education during the coronavirus pandemic. Despite the popularity of online education, not all students are equally successful in asynchronous online courses. The situation has been even worse during the coronavirus pandemic because most students have had no choice but to take their courses online. Dray, Lowenthal, Miszkiewicz, Ruiz-Primo, and Marczynski (2011) indicated that students' personal traits of self-direction and initiative are significant predictors of online learners' success. Recent studies also demonstrate that in order for learners to succeed in online courses, they must have the capacity to regulate their learning (Hew & Cheung, 2014; Kizilcec & Schneider, 2015). With the continuous growth of online courses and online programs offered by higher education, it is important to understand online students' self-regulated learning (SRL) processes so that we can implement strategies to enhance students' self-regulation abilities and thus improve their academic performance.

Although numerous studies about SRL have been conducted in online learning environments, existing research has heavily relied on self-reported surveys (Winne & Perry, 2000). Self-reported data from students have been criticized as lacking validity (Hadwin, Nesbit, Jamieson-Noel, Code, & Winne, 2007; Winne, 2010; Zimmerman, 2008). One possible solution to address this issue is to use learners' trace data collected by learning management systems as a supplement to the self-reported SRL data. Traces are defined as “observable indicators about cognition that students create as they engage with a task” (Winne & Perry, 2000, p. 551). Recent studies (Hwu, 2003; Yu & Zhao, 2015) have indicated that online students' behavioral data are more accurate because the data collected from modern tracking technologies occur in actual learning situations in real-time. Learners may be aware of the data collection taking place, but it is relatively unobtrusive and difficult for learners to alter, so one could assert that more authentic learning behaviors can be recorded on a large scale using this approach. Winne and Perry (2000) proposed two different conceptualizations of SRL: as an aptitude and as an event. Winne (2010) believed that self-reported SRL should be considered as an aptitude and trace data could be treated as an event. Trace data becomes the raw material for researchers to track aptitudes “in action” and how aptitudes may evolve as students make progress in their studies.

The purpose of this study is to investigate whether students' self-reported SRL aligns with their behavior as indicated by the digital trace data collected through the learning management system. We hope that this study will help inform how trace data can be used to enhance online teaching and learning. The research questions posed are as follows:

  • (1)

    How do the digital trace data collected by the learning management system reflect the students' self-reported SRL?

  • (2)

    What is the relationship between students' performance and the digital trace data and self-reported SRL data?

  • (3)

    What are the patterns of learning behaviors based on the digital trace data and self-reported SRL data?

  • (4)

    What are the explanations for any differences between the students' self-reported SRL data and the digital trace data?

Section snippets

Theoretical framework

Several SRL models have been proposed in existing self-regulated learning literature, including Zimmerman's cyclical phases model (Zimmerman, 2000), Pintrich's SRL model (Pintrich, 2000), Boekaerts' dual processing model (Boekaerts, 2011), the Conditions, Operations, Products, Evaluations, and Standards (COPES) model (Winne & Hadwin, 1998), and Efklide's Metacognitive and Affective Model of SRL (MASRL) (Efklides, 2011). SRL is a consistent adaptive process, the momentary trace data collected by

Literature review on SRL-based learning analytics

Given the volume of data that learning management systems collect, it can be difficult to decide which learning behavior variables to focus on. A review of the existing literature provides some guidance. This section will also explore what existing research has been done regarding the alignment between self-reported SRL and the trace data.

Learning behavior variables proposed in this study

Based on a review of the literature, the majority of related studies have not provided a solid pedagogy or theoretical support for why the learning behavioral variables were chosen. Several studies pointed out that it is critical to align the data collection with SRL model (Siadaty, Gasevic, & Hatala, 2016; Yu & Zhao, 2015). In addition, several researchers have argued that when doing learning analytics research, data should be interpreted from the learners' perspective (Ferguson, 2012) and

Methods

Existing research mainly focuses on quantitative data, such as the clickstream data or the interaction between the learner and the content. The current study includes the interaction data between the learner and the content, the learner and other learners, and the learner and the instructor. In addition, both quantitative and qualitative research methods were used in this study because both types of data can support and supplement each other while also serving different purposes. Maxwell (2013)

RQ1: how do the trace data reflect the self-reported SRL data?

The self-reported SRL data were collected twice: once at the beginning of the course (pre-course) and again at the end of the course (post-course). Based on the OSLQ, six perspectives of self-regulated learning data were self-reported by the participants, including goal setting, environment structuring, task strategies, time management, help-seeking, and self-evaluation. The question is whether the students' self-reported SRL data collected before they began the course significantly differ from

Discussion

In summary, the results of this study suggest that digital trace data is more powerful in predicting learners' performance than self-reported SRL data. Learners can be classified into three groups: high self-regulated learners who spend a lot of time and effort in studying, moderate self-regulated learners who spend moderate time and effort in studying but use a lot of study strategies to improve their performance, and low self-regulated learners who spend the least time and effort in studying.

Conclusion and future research

By highlighting the need to explore new ways to measure students' self-regulatory ability, this study found that digital trace data could reflect students' learning accurately. Thus, in the future, digital trace data from LMSs could be used more often in both SRL research and educational research. While using digital trace data to conduct research, we also need to follow ethical rules and protect participants' privacy. In this study, we anonymized the data and all data were stored in

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declarations of interest

The authors declare that they have no conflict of interest.

The data that support the findings of this study are not publicly available due to privacy and ethical restrictions.

Acknowledgements

I thank Dr. Lloyd Rieber at the University of Georgia for his guidance and advice throughout this research project and for his extensive feedback and suggestions on earlier drafts of this manuscript.

References (38)

  • V. Braun et al.

    Using thematic analysis in psychology

    Qualitative Research in Psychology

    (2006)
  • M. Cho et al.

    Exploring online students’ self-regulated learning with self-reported surveys and log files: A data mining approach

    Interactive Learning Environments

    (2017)
  • J. Cohen

    Statistical power analysis for the behavioral sciences

    (1988)
  • K. Colthorpe et al.

    Know thy student! Combining learning analytics and critical reflections to increase understanding of students’ self-regulated learning in an authentic setting

    Journal of Learning Analytics

    (2015)
  • B.J. Dray et al.

    Developing an instrument to assess student readiness for online learning: A validation study

    Distance Education

    (2011)
  • A. Efklides

    Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model

    Educational Psychologist

    (2011)
  • G. Farias et al.

    Teacher as judge or partner: The dilemma for grades versus learning

    Journal of Education for Business

    (2010)
  • R. Ferguson

    Learning analytics: Drivers, developments and challenges

    The International Journal of Technology Enhanced Learning

    (2012)
  • D. Gasevic et al.

    Let’s not forget: Learning analytics are about learning

    Tech Trends

    (2015)
  • 1

    Present address: 219 Van Pelt and Opie Library, Michigan Technological University, Houghton, MI 49931, United States of America.

    View full text