1 Introduction

The start of 2020 brought significant changes, challenging almost all spheres of life. The coronavirus (COVID-19) pandemic made universities across the world mobilise and move all teaching online. Not long ago MOOCs were a source of monetisation, but amid the coronavirus outbreak many MOOC providers offered temporary free access to MOOCs, helping universities teach remotely (Schwartz 2020). Although teaching online might seem like a good solution in such a situation, the old problems of MOOCs, which have been identified but not solved, could become a potential danger to its successful realisation.

One of the biggest challenges, faced by MOOCs, is the high percentage of learners who do not reach the finish line (Kizilcec et al. 2013; Perna et al. 2014; Reich and Ruipérez-Valiente 2019). Statistics show that the dropout rate can be up to 90–98% on various online platforms (Rivard 2013; Perna et al. 2014; Reich 2014; Healy 2017). Researchers specify diverse factors which have an impact on successful MOOC completion, for example, young people, the so-called ‘net generation’, are considered more digitally adapted, and they show higher chances of finishing online courses compared to an older generation who needs more guidance (Bennett et al. 2008). MOOC learners’ motivation to learn and develop personal and professional identity is also related to course completion (Yuan and Bowel 2013). Individuals who are self-regulated in their learning tend to achieve more positive academic outcomes than individuals who do not exhibit self-regulated learning behaviors (Barnard et al. 2010; Maldonado-Mahauad et al. 2018). The research conducted to date proves self-regulated learning (SRL) to be the most essential skill to succeed in MOOCs (Littlejohn et al. 2016). Barnard et al. (2010) describe self-regulated learners as those who are able to set their academic goals, manage their time, seek help from their peers and instructors when needed, monitor their work, evaluate their academic progress, and create a productive environment for learning. The value of self-regulated learning skills was first attributed to the traditional offline academic environment (Zimmerman 1990), and then this idea was brought to the online learning environment (Barnard et al. 2009; Milligan et al. 2013; Milligan and Littlejohn 2014; Fontana et al. 2015). As research shows, there is a significant relationship between learners’ self-regulated learning skills and MOOC completion (Milligan et al. 2013).

Barnard et al. (2009) conceptualise SRL as a complex construct consisting of six dimensions: environment structuring, goal setting, time management, help-seeking, task strategies, and self-evaluation. Although there are many studies which confirm the importance of different dimensions of self-regulated learning for MOOC completion, we argue that not all of them can be applied to the MOOC environment. In particular, help-seeking, which implies face-to-face or online meetings with classmates and getting help from the instructor through email, does not seem to be relevant for the MOOC context. By design MOOCs have limited interaction between learners and instructors. As Baker et al. (2018) find, only 7% of MOOC learners received feedback from instructors. It was also shown that the level of activity of MOOC learners’ on online platforms is low: 90% of learner activity includes the review of the same information (Breslow et al. 2013) and 94% of them do not take part in online discussions (Qiu et al. 2016). Despite the fact that researchers highlight the importance of learner-instructor interaction to increase distant learners’ motivation and satisfaction (Moore 1989), only learner-content interaction has been currently implemented to its full potential (Gameel 2017). Therefore our study fills the gap in understanding the structure of self-regulated learning skills in the MOOC environment utilising the Online Self-Regulated Learning Questionnaire (OSLQ).

2 Background

2.1 Self-regulated learning in the online settings

An increasing number of studies investigate the role of self-regulated skills in online learning (Bernacki et al. 2011; Wang et al. 2013). In self-paced, open-ended, and non-linear learning environments students gain autonomy, but at the same time this adds responsibility towards their learning process. Self-regulated learning skills are critically important in online-environment because students should plan, manage and control their learning activities in order to finish courses successfully (Wang et al. 2013). The ability to self-regulate the learning process helps achieve personal objectives in MOOCs: goal setting and strategic planning are seen as strong predictors of goal attainment (Kizilcec et al. 2017). Interviews conducted with MOOC students indicate that high SRL learners tend to connect their online education experience to their personal needs, such as career advancement (Littlejohn et al. 2016). The research by Littlejohn et al. (2016) demonstrated that students with high level of SRL utilize a more flexible approach to organize their learning process. For example, high self-regulated students can spend more time watching video lectures and submitting weekly tests, they also are more likely to revisit course materials (Kizilcec et al. 2017).

Interaction between learners and instructors is also deemed to be a vital element of SRL. The results of the study by Sunar et al. (2017) suggest that dropout rates will be lower when learners engage in repeated and frequent social interactions. The analysis of the discussions in an eight week FutureLearn MOOC with 9855 participants revealed that if a learner starts following someone, and if learners also interact with those they follow, the probability of their finishing the course is higher. Feelings of isolation and having no one to ask for help increase chances of failing a MOOC (Khalil and Ebner 2014). Despite the widely-accepted importance of social engagement, a MOOC environment does not allow participants to interact easily with other learners and instructors for better social engagement. MOOC instructors cannot invest much time to take care of a large number of students, their attention to each student is limited. However, this does not make social engagement less important, rather places some barriers on success of MOOCs.

2.2 Validity of the online self-regulated learning questionnaire

The Online Self-Regulated Learning Questionnaire (OSLQ) has become one of the widely-used instruments to assess self-regulated learning skills in the online environment (Barnard et al. 2009). The OSLQ was used in both cross-sectional (e.g. Kintu and Zhu 2016; Onah and Sinclair 2016) and longitudinal (Tabuenca et al. 2015) studies. Also, the OSLQ was adapted into the Turkish (Korkmaz and Kaya 2012), Romanian (Cazan 2014), Russian (Martinez-Lopez et al. 2017) and Chinese (Fung et al. 2018) languages. However, to the best of our knowledge, none of the previous studies managed to provide comprehensive evidence of the validity of OSLQ. We assume that self-regulated learning in the MOOC settings is different from the one suggested by Barnard et al. (2009). Our assumption can be explained based on the following.

First, studies attempted to test construct-related validity of the OSLQ using Confirmatory Factor Analysis (CFA) (Barnard et al. 2009; Korkmaz and Kaya 2012; Martinez-Lopez et al. 2017; Fung et al. 2018) utilise rather small samples. None of the samples used in the studies meets the minimum criterion of 10 participants per estimated parameter (Schreiber et al. 2006). There are 54 parameters that need to be estimated to test the structure of the OSLQ. This indicates that a minimum sample size required for CFA is 540 participants. Second, CFA standardised coefficients for the help-seeking dimension reported in Barnard et al. (2009) were much lower for online students than for blended course students. Another potential problem is the low standardised coefficients (< .40) for the factor ‘help-seeking’ presented in the CFA model for online students. Third, there is only one study (Martinez-Lopez et al. 2017), which utilised a sample of MOOCs learners to test the validity of the OSLQ, while the majority of studies were based on either traditional or blended samples. Finally, criterion-related validity was tested as correlation between the OSLQ’s subscales and academic achievements (Cazan 2014). According to the results, the association of two subscales (goal setting and environment structuring) with academic performance was statistically significant. However, no evidence for the association of the OSLQ’s total score and educational outcomes was presented. Table 1 summarises the methods used in prior studies to support the OSLQ validity.

Table 1 Summary of the OSLQ validity studies
Full size table

After reviewing research designs, methods and results from previous studies, we can conclude that there is no strong support for both construct and criterion validity of the OSLQ. Therefore, self-regulated learning might have a different structure from the one proposed by Barnard et al. (2009). Based on the studied literature, we hypothesise that help-seeking subscale might not be useful in assessing SRL skills among MOOC learners, and the five-factor hierarchical model is more appropriate instead of the six-factor hierarchical model for the current MOOC environment.

3 Data and methods

3.1 Administrative procedure

The sample consists of MOOCs learners, who were enrolled in online courses on the National Open Education Platform (NOEP)Footnote 1 in 2017, and previously agreed to receive e-newsletters from the platform. NOEP markets itself as a Russian analogue to MOOC providers like Coursera or edX which offers various courses mainly in the Russian language. Each course, posted on the platform, is subject to expert analysis. Currently, there are 582 MOOCs created by 16 leading Russian universities. Learners received an email invitation to participate in a web-based online survey and share their experience with MOOCs. The survey was programmed with the EnjoySurvey software. It was anonymous and the participation was voluntary. As this study was targeted at the student population, we filtered out everybody who did not specify himself or herself as a student currently enrolled at university.

3.2 Descriptive statistics of the sample of MOOC learners

The total number of qualified respondents is 913. We kept only those respondents who completed the questionnaire. The sample included 68% of female students, the average age is within the range 19–22 years and the majority of the respondents are pursuing their Bachelor degree. Learners who participated in the survey were enrolled in a wide range of MOOCs. The most popular MOOCs, which attracted the largest number of learners, were ‘Philosophy’, ‘Marketing’, ‘Programming’, and ‘Game Theory’. The detailed descriptive statistics of the sample is presented in Table 2.

Table 2 Socio-demographic characteristics of the sample
Full size table

The majority of students in the sample (63%) were enrolled in 1–2 MOOCs in 2017 (see Table 3).

Table 3 The number of students enrolled on MOOCs in 2017
Full size table

3.3 Instrument adaptation procedure

The use of self-regulation strategies was measured by the OSLQ. The original English version of the OSLQ was adapted to the Russian language according to the ITC Guidelines for Translating and Adapting Tests (International Test Commission 2017). Back-translation procedures were adopted to ensure conceptual equivalence across languages. Two translators who are fluent in both the source language (English) and the target language (Russian) and had previous experience in translating survey instruments were selected. One translator did the translation of the original version into Russian and a back translation into English was performed by another translator. The content validity was assessed based on experts’ opinion as to whether or not the instrument is measuring what it is supposed to measure. The group of experts (8 people) who were part of the research group with the focus on online learning discussed the adapted version in the format of a focus group. Their aim was to obtain evidence of such features of the instrument as sufficiency, clarity, relevance, and the match between the items and the definition of the construct controlling for possible biases, for example, gender, culture, and age (Goodwin and Leech 2003). After the structured procedure, it was decided to remove the following items from the OSLQ:

  1. 1.

    Goal setting: I set goals to help me manage studying time for my online courses.

  2. 2.

    Task strategies: I read aloud instructional materials posted online to fight against distractions.

  3. 3.

    Self-evaluation: I summarise my learning in online courses to examine my understanding of what I have learned.

  4. 4.

    Self-evaluation: I ask myself a lot of questions about the course material when studying for an online course.

These four items were deemed by the experts as confusing because of their vague meaning in the Russian language and logical discrepancies. For example, the meaning of the item ‘I set goals to help me manage studying time for my online courses’ is similar to the item ‘I set short-term (daily or weekly) goals as well as long-term goals (monthly or for the semester)’. The omitted ‘self-evaluation’ items are different in essence from the two that were kept. We kept the items which describe students’ communication with their classmates and evaluation of their progress, whereas the two deleted items were considered by the experts more of a monitoring strategy in which the student summarises or questions oneself. The item ‘I read aloud instructional materials posted online to fight against distractions’ was deleted as the research shows that the strategy of reading instructions aloud to get focused and comprehend information works for kids, school children with attention deficit hyperactivity disorder (US Department of Education 2008) and our target audience was university students.

3.4 Analytical strategy

3.4.1 Methods

The data analysis was performed in two stages. The first stage was aimed to check the OSLQ’s construct-related validity. In the first stage, we tested the structure of the questionnaire utilising Confirmatory Factor Analysis (CFA). We also evaluated the internal consistency of the six subscales using the classical reliability index (Cronbach’s alpha). The second stage was focused on assessing the OSLQ’s criterion-related validity using independent samples t-tests comparison.

3.4.2 Construct-related validity

We applied CFA to test the structure of the questionnaire. We tested two CFA models: the original six-factor hierarchical or second-order model suggested for the OSLQ in Barnard et al. (2009) and the alternative five-factor hierarchical model, since we hypothesise that help-seeking subscale might not be useful in assessing SRL skills among MOOC learners.

A hierarchical or a second-order model suggests that there is a higher-order factor and the lower-order factors (Chen et al. 2012). In the case of the OSLQ, self-regulated learning is the higher order-factor, which accounts for the commonality between the six subscales (goal setting, environment structuring, task strategies, time management, help-seeking, and self-evaluation). The correlation matrix for the 20 items of the OSLQ is presented in Table 4. See Table 5 for the correlations between the first-order subscales.

Table 4 Correlation matrix of the OSLQ items
Full size table
Table 5 Correlation matrix of the OSLQ subscales
Full size table

The correlations between factors are not high (< .70), suggesting that each of the subscales forms a separate construct. The correlation matrix of the OSLQ items shows no significant correlations exceeding .70 between items from different factors.

The STATA 15.0 software package was used for CFA. We chose maximum likelihood (ML) estimation because there was no evidence of excessive non-normality.

First, we estimated the fit between the model and the observed data. Three conventional statistics, reflecting the model fit, were reported: the root mean square error of approximation (RMSEA), the Tucker Lewis Index (TLI), and the Comparative Fit Index (CFI). We relied on the following values of statistics which indicated the acceptable fit (Byrne 2010; Schreiber et al. 2006): RMSEA close or below .06 (.08) with confidence interval, TLI and CFI close or above .90 (.95). No post hoc model modifications were made.

Second, we calculated the standardised path coefficients from the latent variable constructs to the items, and from the higher order construct to the latent variable constructs.

Third, we determined which model better fits the data. We compared them using two fit indexes: Akaike information criterion (AIC) and Bayes information criterion (BIC). According to Schreiber et al. (2006), the smaller these indexes are, the better the model fits the data.

3.5 Criterion-related validity

One of the commonly used approaches in criterion-related validity research is group-comparison. It usually aims to investigate differences between instruments scores across groups of examinees (Goodwin and Leech 2003). A comparison variable should be related to the measuring construct. In this case, we rely on prior studies about MOOC completion. The research indicated that learners with a high level of SRL skills perform better than their lower counterparts (Kizilcec et al. 2017; Milligan et al. 2013). It has been demonstrated that learners, who set goals to finish the course, are less likely to drop out (Handoko et al. 2019).

To assess the criterion-related validity of the OSLQ, we used students’ plans to finish the course (the question ‘Did you plan to receive the verified certificate at the end of the course?’). We divided students into two groups according to their plans. The first group of students informed us that they planned to receive a verified certificate, the second group did not. Their plan to receive a verified certificate is perceived as the criterion variable. We performed an independent samples t-test to compare mean scores on SRL between these groups.

4 Results

4.1 Construct-related validity

Based on the previous research by Barnard et al. (2009), we tested the original six-factor hierarchical model suggested for the OSLQ. The model was specified according to the following criteria: (1) each item had a nonzero loading on the six first-order factors; (2) standard errors associated with each item were uncorrelated; (3) covariance between six first-order factors was explained by a second-order factor – self-regulated learning.

Figure 1 provides the standardised path coefficients from the latent variable constructs to the items and from the higher order construct to the latent variable constructs. The values of fit-statistics indicate an unacceptable fit between the model and the observed data, RMSEA = .08 (.08; .09), CFI = .88, and TLI = .86. Each of the fit indexes exceeds the cutoff criteria suggested by Byrne (2010) and Schreiber et al. (2006). The results do not support the original six-factor hierarchical model.

Fig. 1
figure 1

The six-factor hierarchical model (N = 913). Note. χ2(164) = 1200.68, p = .00; RMSEA = .08 (.08; .09), CFI = .88, and TLI = .86. *** p < .001.

Full size image

The six-factor hierarchical model was modified by removing the help-seeking subscale from the model. Relying on the results of the mentioned research suggested that help-seeking behaviour might be not common among MOOC learners, we hypothesise that a five factor hierarchical model could better fit our data. The model was specified according to the following criteria: (1) each item had a nonzero loading on the five first-order factors; (2) standard errors associated with each item were uncorrelated; (3) covariance between five first-order factors was explained by a second-order factor – self-regulated learning.

The values of fit-statistics indicate an acceptable fit between the five factor hierarchical model and the observed data, RMSEA = .07 (.06; .07), CFI = .94, and TLI = .93. Figure 2 presents the standardised path coefficients from the latent variable constructs to the items and between the constructs. The paths in the second model are all significant (p < .001) with standardised values ranging from .59 to .92 from the first-order factors to the items, and standardised values ranging from .45 to .94 from the second-order factor to the first-order factors.

Fig. 2
figure 2

The five factor hierarchical model (N = 913). Note.: χ2(99) = 513.09, p = .00; RMSEA = .07 (.06; .07), CFI = .94, and TLI = .93. *** p < .001.

Full size image

Next, we determined which model better fits the data. Table 6 provides the fit indexes for each of the models. AIC and BIC for the five-factor hierarchical model are smaller: 53,558.51 > 42,103.73 and 53,876.41 > 42,359.02. According to these results, the second model better fits the data. The five-factor hierarchical model gives a better approximation and interpretation of our data about SRL behaviour among MOOC learners. Therefore, we suggest a redefined model of the OSLQ for MOOC learners, because help-seeking skills appear not relevant in this environment.

Table 6 Comparison of model fit indices
Full size table

The average classical reliability (Cronbach’s α) ranges from 0.72 to 0.90 for the subscales (Table 7). Mean Cronbach’s α for the six-factor model is 0.89, for the five-factor model is 0.88. High reliability coefficients (α > 0.80) indicate that the questionnaire can be used for both research and diagnostic purposes (Tavakol and Dennick 2011).

Table 7 Reliability coefficients
Full size table

4.2 Criterion-related validity

The research shows that successful MOOC learners tend to have higher scores on SRL skills (Milligan et al. 2013; Kizilcec et al. 2017). Since we did not have access to the data about students’ grades, we asked them whether they planned to obtain the verified certificate for MOOC completion. Students’ answers for the question ‘Did you plan to receive the verified certificate at the end of the course?’ was used as a comparison variable. The independent samples t-tests for each dimension were conducted to determine if there is a difference in SRL skills between students, which planned to obtain the verified certificate, and which did not (Table 8).

Table 8 The independent samples t tests results
Full size table

According to the results, the mean on all of the six OSLQ’s subscales statistically differs among the two groups of students. There are statistically significant differences, at .001 or .01 levels of significance, between students who planned to obtain the verified certificate, and those who did not. The results show that students, who planned to receive the certificate, score higher on goal setting, environment structuring, task strategies, time management, help-seeking, and self-evaluation. We find that the difference in average on the OSLQ subscales ranges from 0.21 standard deviations for ‘task strategies’ to .39 standard deviations for ‘goal setting’. Despite the fact that ‘help-seeking’ dimension might be redundant in MOOCs, it still determines higher educational outcomes.

5 Discussion and conclusions

The main finding of this study is that the original six-factor model for OSLQ is not always effective and the Russian version of the OSLQ has a five-factor hierarchical structure with the second-order factor SRL in the MOOC environment. The results suggest that the OSLQ can be used to measure people’s ability to set goals, structure the online environment, manage time, and evaluate their progress in the online environment. However, the subscale ‘help-seeking’ does not work in the current context of MOOCs.

The poor statistics and logical incoherence of the items from the dimension ‘help-seeking’ points to one of the main challenges faced by MOOCs – low communication between MOOCs’ students and instructors during the learning process. Researchers demonstrate that 94% of learners do not participate in online discussions (Qiu et al. 2016), and 90% of forum activity is revisiting previous threads (Breslow et al. 2013). In addition, one of the reasons learners withdraw from MOOCs is the failure to understand the content and having no one to turn to for help (Belanger and Thornton 2013). According to Mahasneh et al. (2012), more than half of students (64%) avoid help-seeking, even if they do not understand the material. The researchers explain students’ fear of asking for help as not wanting to be seen as incompetent (Ryan et al. 2001).

The pandemic outbreak in the world, which stopped normal face-to-face classes, shook the whole education system. This suggests online platforms and instructors need to be prepared to embrace this challenge by improving the quality of platforms and developing new pedagogical practices. As educators and researchers state, education is not only a question of good presentation and design, such elements as support and guidance are also essential (Shaughnessy et al. 2010). Eventually, online education has to become an old-new product with different instructor-learner relations and different ways of using information.

Although this study shows important results, it has some limitations. The results were derived from the sample of Russian MOOC learners. As previous studies show, self-regulated learning skills and strategies can be dependent on the environment (Schunk 2001; Barnard et al. 2010) and can function differently in various countries. Therefore, the results may not be generalized to the whole population of MOOC learners. Future research should consider verifying whether the OSLQ works similarly in different cultural contexts. For this purpose, first, comprehensive evidence of the validity of OSLQ in other cultural settings should be found following strict statistical requirements (for example, homogeneous group of respondents, appropriate sample size). Second, Differential Item Functioning (DIF) analysis can be utilised. DIF appears when there are different probabilities of endorsing a given item on a multi-item scale although the measured characteristics are the same (Zumbo 1999).

Further research could analyse convergent validity of the OSLQ through taking two measures that are supposed to be measuring the same construct and show that they are related. In other words, the correlation between OSLQ scores and other instruments, for example, the Academic Self-Regulation Scale (Magno 2010), the Motivated Strategies for Learning Questionnaire (Pintrich et al. 1993) can be investigated. Predictive validity can be assessed through a comparison of the OSLQ scores with a degree of completion of MOOCs taken right after measuring self-regulated learning skills. Finally, investigating heterogeneous samples, not limited to one category of respondents, would show whether our results are valid for other types of MOOC learners.