Beyond positive and negative emotions: Looking into the role of achievement emotions in discussion forums of MOOCs

https://doi.org/10.1016/j.iheduc.2019.100690Get rights and content

Highlights

  • This paper quantifies the effect of achievement emotions on student dropout.

  • This paper introduces large-scale data mining to study emotions.

  • This paper helps design a support mechanism to enhance student commitment.

Abstract

Millions of students register for Massive Open Online Courses (MOOCs) to look for opportunities for learning and self-development. However, the learning process usually involves emotional experience, which may affect students' participation in the course, eventually resulting in dropping out along the way. In this study, we quantify this effect. Particularly, this research goes beyond focusing on only the single dimension of positive or negative emotions as many prior studies do. Instead, informed by the control-value theory, a more integrated framework of achievement emotions is applied in order to gain a comprehensive understanding of the role of emotions in MOOC students' learning experiences. Specifically, we first built and validated a machine learning model to automatically detect the achievement emotions in the forum posts. Then survival analysis was used to quantify the influence of achievement emotions on student dropout. The results show a different influencing mechanism for expressed and exposed achievement emotions on student survival in the MOOC course. Implications of the results are then discussed in terms of intervention design to improve student retention in MOOCs.

Introduction

With the recent development in open educational resources in both industry and academia, Massive Open Online Courses (MOOCs) have taken the center stage of discussion especially in the higher education sector (Reich, 2015; Wautelet et al., 2016). MOOCs can enable thousands of students to take courses at their convenience without cost or at low cost. It is largely because they provide a specific means for enabling more equitable access to learning that MOOCs have gained popularity in recent years (Tang and Carr-Chellman, 2016). In spite of all their potential, a significant concern about MOOCs is the extremely high attrition rate (approximately 90%) that has been reported (Hew & Cheung, 2014). Such a high dropout rate has often been cited as a scale-efficacy tradeoff (Onah, Sinclair, & Boyatt, 2014). While the reasons for dropping out are diverse in MOOCs, studies have shown that students' emotions evidenced in their learning process can significantly affect students' continued participation (Dillon et al., 2016; Dillon et al., 2016; Hillaire, Iniesto, & Rienties, 2017).

Understanding the role of emotion becomes more important especially if we consider the gradual nature of attrition in MOOCs. Much of the research on MOOC dropouts centers on the summative metric of attribution, e.g., through conducting correlational analysis between dropouts with click-stream evidence of engagement (Ramesh, Goldwasser, Huang, Daumé III, & Getoor, 2013) or building dropout prediction models (Halawa, Greene, & Mitchell, 2014). However, several seminal works have shown that attrition takes place over time (Whitehill, Mohan, Seaton, Rosen, & Tingley, 2017; Yang, Wen, Howley, Kraut, & Rose, 2015). That is, while many participants either never engage in the course at all or drop out after the first week, a significant portion of participants remains in the course for several weeks and then drops out along the way (Yang, Sinha, Adamson, & Rosé, 2013). This suggests that there are students struggling to stay involved. Understanding the participation of struggling students as they struggle and ultimately quit the course can help find potential ways to provide appropriate intervention and scaffolds to support them. Investigations on struggling students from an emotional perspective are essential to understand their participation and learning experience. Emotions have been constantly identified as one of the key factors in influencing online learning commitment (Lee & Choi, 2011; Pillay, Irving, & Tones, 2007). Supporting these struggling students from the emotional angle may be the first low hanging fruit to improve the retention rate of a MOOC.

Emotion has a complex influence on learning and commitment. Positive emotion experienced during the learning process is not necessarily related with longer commitment in MOOCs and negative emotion may lead to a beneficial effect on learning outcomes (Barak, Watted, & Haick, 2016; Lee & Choi, 2011). In fact, emotions are more than a simple dichotomy of positive and negative emotions. In learning and academic settings, the kind of emotions that really matters to the commitment and outcomes are achievement emotions (Pekrun, 2006). Achievement emotion can be further deconstructed as positive activating, positive deactivating, negative activating and negative deactivating emotions according to the control-value theory (Pekrun, 2006). As an integrative framework to approach students' learning experiences, the control-value theory of achievement emotions may serve as the basis for instructors to design interventions to improve learning engagement in MOOCs. By contrast, research previously investigated the influence of the single dimension of emotions (positive or negative emotions) on attrition overtime (Dillon, Ambrose, et al., 2016; Dillon, Bosch, et al., 2016), without revealing the mechanism for different achievement emotions on student attrition.

This work focused on exploring achievement emotions and their complex impact on student dropouts in MOOCs. It aimed to understand how achievement emotions influence learner survival as they struggle and ultimately withdraw from a MOOC forum. Relying on an actual MOOC dataset, we first depicted a classifier that automatically predicted the different achievement emotions of students' posts in the MOOC forum. Then with reasonable accuracy, the built classification model was applied to all the posts to identify the four achievement emotions. Finally, a survival modeling technique was used to quantify the effect of different achievement emotions on student attrition longitudinally. The rest of the paper is organized as follows: we begin by discussing the literature and theoretical framework underlying this study. Next, we describe the dataset and the methodology used. Then, our analysis and results are presented. Finally, we discuss the results and summarize this study by anchoring it to the prior work.

Section snippets

Dropout analysis in MOOCs

The hallmark of MOOCs is that enrolled learners are involved in the social learning with virtual unlimited peers (Reich, 2015). Participation in the course is the prerequisite for learners to benefit from the large-scale social learning. Conversely, the large number of attritions becomes the tradeoff to extend educational resources to the masses (Onah et al., 2014). Therefore, improving the retention rate of MOOCs has become a major focus of recent research.

Many studies explored the

Research dataset and context

The dataset used in this research was derived from a Coursera MOOC offered by a large public research university in the United States. This course was selected because 1) this was a highly valued MOOC by many online portals and students; 2) its course design (e.g., course format, course length, discussion forum settings) and student population (e.g., number, profile) was exemplary (Bonafini, Chae, Park, & Jablokow, 2017). With that being said, empirical findings resulting from this dataset were

Achievement emotion detection results

Using different features and various algorithms, machine learning models were built to automatically identify different kinds of achievement emotions in MOOC forum posts. Table 3 shows the results for the performance of the machine learning models. By comparing the ROC_AUC and Kappa values, SVM has the highest performance with ROC_AUC (0.91) and Kappa (0.61) when using all the language summary features, linguistic features, and the LDA topic features. The prediction performance of the machine

Discussion and conclusion

Increasing the retention rates of MOOCs is tremendously important for their future adoption. In this study, we looked beyond the dichotomy of only positive and negative emotions, using an integrative framework of achievement emotions to examine their influence on students' commitment in MOOCs. An accurate machine learning model was built to automatically detect the achievement emotions in the forum posts. Then survival analysis was applied to quantify the effect of achievement emotions on

Declaration of Competing Interest

The authors declare that they have no conflict of interests.

Wanli Xing is an Assistant Professor in Instructional Technology at Texas Tech University, USA with background in learning sciences, statistics, computer science and mathematical modeling. His research interests are educational data mining, learning analytics, and CSCL.

References (54)

  • S.G. Barsade et al.

    Why does affect matter in organizations?

    Academy of Management Perspectives

    (2007)
  • F. Bonafini et al.

    How much does student engagement with videos and forums in a MOOC affect their achievement

    Online Learning Journal

    (2017)
  • L.R. Buis

    Emotional and informational support messages in an online hospice support community

    CIN: Computers, Informatics, Nursing

    (2008)
  • R.A. Calvo et al.

    Affect detection: An interdisciplinary review of models, methods, and their applications

    IEEE Transactions on Affective Computing

    (2010)
  • Y. Chen et al.

    Facilitating Students’ Interaction in MOOCs through Timeline-Anchored Discussion

    International Journal of Human-Computer Interaction

    (2019)
  • M.K. Dalal et al.

    Automatic text classification: A technical review

    International Journal of Computer Applications

    (2011)
  • L.M. Daniels et al.

    A longitudinal analysis of achievement goals: From affective antecedents to emotional effects and achievement outcomes

    Journal of Educational Psychology

    (2009)
  • J. Dillon et al.

    Student affect during learning with a MOOC

  • J. Dillon et al.

    Student emotion, co-occurrence, and dropout in a MOOC context

  • S. Halawa et al.

    Dropout prediction in MOOCs using learner activity features

  • J.T. Hancock et al.

    I'm sad you're sad: Emotional contagion in CMC

  • G. Hillaire et al.

    Toward emotionally accessible massive open online courses (MOOCs)

  • A. Jarrell et al.

    The link between achievement emotions, appraisals, and task performance: Pedagogical considerations for emotions in CBLEs

    Journal of Computers in Education

    (2016)
  • A. Khan et al.

    A review of machine learning algorithms for text-documents classification

    Journal of Advances in Information Technology

    (2010)
  • J.P. Klein et al.

    Survival analysis: Techniques for censored and truncated data

    (2005)
  • R. Kop

    The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course

    The International Review of Research in Open and Distributed Learning

    (2011)
  • S.B. Kotsiantis

    Supervised machine learning: A review of classification techniques

  • Cited by (76)

    • From the learner's perspective: A systematic review of MOOC learner experiences (2008–2021)

      2022, Computers and Education
      Citation Excerpt :

      An earlier work by Hew (2016) provided a list of strategies that students found engaging, including active learning and peer interaction (p. 328), elements the integration of various technological tools is designed to facilitate. In other instances, technological tools can track elements of learners' engagement in MOOCs (Sun et al., 2020; Xing et al., 2019) and facilitate learners' hesitancy to interact in MOOCs. Bozkurt et al. (2018) discussed the use of a bot in MOOCs to help connect learners, and several learners learners responded to positively, a finding in keeping with another item on the list of what students find engaging in courses: instructor accessibility (Hew, 2016).

    • Mining opinions on LMOOCs: Sentiment and content analyses of Chinese students’ comments in discussion forums

      2022, System
      Citation Excerpt :

      This indicates that MOOCs have acquired a significant position in China's higher education. The prominent roles of MOOCs have attracted increased research attention, which mainly centers on several areas: types of MOOCs (Downes, 2012; Hew et al., 2018; Teixeira & Mota, 2014), the impact of MOOCs on institutions (Lombardi, 2013; Motzo & Proudfoot, 2017; O'Connor, 2014), instructors' motivation and challenges in teaching MOOCs (Deng et al., 2017; Hew & Cheung, 2014), students' motivation to register MOOCs, factors influencing course completion, and perceptions of MOOCs (Deng et al., 2019; Hew & Cheung, 2014; Li et al., 2022; Wang et al., 2021; Wen et al., 2014; Xing et al., 2019), and analysis of log data of MOOCs (de Barba et al., 2016; Liu et al., 2018; Wen et al., 2014). MOOCs are generally classified into cMOOCs and xMOOCs (Downes, 2012; Teixeira & Mota, 2014).

    View all citing articles on Scopus

    Wanli Xing is an Assistant Professor in Instructional Technology at Texas Tech University, USA with background in learning sciences, statistics, computer science and mathematical modeling. His research interests are educational data mining, learning analytics, and CSCL.

    Hengtao Tang is an Assistant Professor in Educational Technology at the University of South Carolina. His research interests include learning analytics, CSCL, self-regulated learning, STEM education, and knowledge structure.

    Bo Pei is a doctoral student in Instructional Technology at Texas Tech University, USA. His research interests are educational data mining, learning analytics, and CSCL.

    View full text