Introduction

In recent decades, one of the main educational milestones is the advent of a new form of learning called e-learning (electronic learning), based on the use of the internet and technology to support students’ online education. Nowadays, this form of learning is becoming particularly important due to the limitations defined by the authorities to restrain the spread of pandemics such as the one caused by Covid-19. The use of e-learning poses important advantages including the enabling of a more flexible temporal and spatial interaction than other forms of learning. Besides, vast amounts of learning process data can be collected, since it is based on the use of Learning Management Systems (LMS). Moodle (Dougiamas & Taylor, 2008) is one of the most used LMS overall, because, among other advantages, it is free, open and there is an important community of users who support its development. Data recorded by Moodle, in particular those that reflect students’ interactions with educational resources, can be of great interest and applicability for building student behavior models. To analyze these data, approaches such as Educational Data Mining (EDM) and Learning Analytics (LA) are useful (Romero et al., 2008). In EDM, a field whose purpose is the extraction of knowledge from educational data, there are well-defined problems that have been addressed by the scientific community, such as the prediction of students’ performance (Romero & Ventura, 2013, 2020). Recently, it is more frequent to find works that propose new approaches to analyzing educational data for a particular course. However, one of the due challenges is creating models for a particular course that can be useful when used in other courses (Baker, 2019). These are what we call transferable or portable models (Boyer & Veeramachaneni, 2015).

In our previous work (López-Zambrano et al., 2020), we obtained models generated from Moodle’s logs data and we studied the degree of portability of the models between subjects, grouped by area of knowledge and by the usage level of platform resources. We used Moodle’s native raw attributes which, in certain combinations of courses, led us to a certain loss in the portability of models since these low-level attributes are very dependent on each particular course. To overcome this limitation from our previous research, in this paper we present a new approach based on the use of resources from the semantic web area, in particular, ontologies (Fong et al., 2011; Tang & Fong, 2010). One of the most promising lines in this respect, particularly when analyzing logs of students’ interactions with the LMS, is the categorization or taxonomy of attributes. In this regard, Bloom’s taxonomy plays an important role. Bloom’s taxonomy is a multi-tiered model of classifying thinking according to six cognitive levels of complexity which in this new version are: Remembering, Understanding, Applying, Analysing, Evaluating and Creating (Forehand, 2005). Based on this idea, some works have even defined correspondence between the levels of Bloom’s taxonomy and the different actions conducted by students in Moodle (Rollins, 2010). Some authors (Cerezo et al., 2020) proposed a categorization of low-level attributes into different higher-level codifications, such as Executing, Planning, Learning, and Reviewing. Precisely, our research aims to evaluate the degree of portability of models built by using ontologies of interaction-with-the-platform attributes. To do so, we defined an ontology inspired by Bloom’s taxonomy and based on the work by Cerezo et al. (2020), with the purpose of conducting a comprehensive study to measure the degree of portability of the models built based on that ontology (denoted as ontological models), compared with a previous similar study conducted by the authors López-Zambrano et al. (2020) in which we did not use ontologies but instead employed low-level Moodle attributes (denoted as non-ontological models). The models have been built from students’ interactions with Moodle logs and the class attribute to predict is binary and represents whether or not the student will pass the course (Pass/Fail). In this work, the courses have been grouped according to the usage level of Moodle activities/resources. This approach has already been used in previous studies with satisfactory results (López-Zambrano et al., 2020). Taking all this into consideration, the global objective of this paper is to provide an answer for the research question below:

  • Can the ontological models obtained in one (source) course be applied in other different (target) courses with a similar usage level without losing prediction accuracy?

The rest of the paper is organized as follows: Sect. “Background” reviews the literature related to this research. Section “Materials and methods” describes the data and the experiments. Section “Results” includes and discusses the results obtained. Finally, Sect. “Conclusions” presents the conclusions and future lines of research.

Background

Achieving generalizable and portable models is still an important challenge in the area of EDM, in spite of the important advances made in the last few years (Boyer & Veeramachaneni, 2015; Ding et al., 2019; Gašević et al., 2016; Hunt et al., 2017). In fact, Baker (2019) has considered what he calls the “Generalizability” or “New York City and Marfa” problem as one of the main challenges for the future of EDM, which is explained in detail in López-Zambrano et al. (2020).

To address this challenge, the resource of resources from the semantic web seems to be a promising line. The semantic web is an extension of the current web in which information is provided with a certain meaning, which makes cooperation and portability easier (Dhuria & Chawla, 2014). Fundamental resources from the semantic web are the ontologies, because they provide a common understanding of a domain. In particular, they may be interesting resources in the e-learning field (Al-Yahya et al., 2015).

In this regard, several particular works should be highlighted. In Octaviani et al. (2015) they present a tool, called RDB2Onto, for creating ontologies from Moodle logs, but this work does not validate the utility of such an ontology. In Castro and Alonso (2011) they propose a general architecture for EDM in which there is an educational ontology, but they do not define or develop the ontology, only providing a general statement of it as a part of a higher-level architecture. There are even some works such as the one presented in Chang et al. (2020) where they utilize data mining techniques (association in this case) to build ontology-driven tutoring models for intelligent tutoring systems (this is precisely the opposite process to ours since we use the ontology for a further data mining analysis).

These previous works present general approaches. Other more specific works bear greater similarity to our study because they define particular ontologies to facilitate the EDM process. We found some works where the ontology created is not focused on attributes of students’ interaction with the LMS. In Marinho et al. (2010) they propose an ontology to model EDM tasks, techniques, and parameters. In Grivokostopoulou et al. (2014) they propose an educational system that utilizes ontologies and semantic rules to enhance the quality of educational content (curriculum) and the learning activities delivered to each student. In Nouira et al. (2019), they propose an ontological model for assessment analytics. And finally; in Dorça et al. (2017), they present an approach for the automatic and dynamic analysis of learning object repositories in which ontology models the relationships between the attributes and learning styles of the learning objects.

Other related works are those that define ontologies to model data of students’ interactions with LMS resources. In El-Rady (2020), they propose an ontology where the student is the main class from which a series of associations arise that are connected to other classes that model the students’ data (education, profile, social activities, etc.). That ontology is used as a part of a validation process to predict student dropout rates. Other related works are based on the idea of organizing the interaction attributes as part of a kind of taxonomy. It is worth highlighting the work presented in Cerezo et al. (2020), where they propose a process mining method for a self-regulated learning assessment, and make use of an ontology inspired by Bloom’s taxonomy. In Montenegro-Marin et al. (2011), they also propose an ontology based on the idea of taxonomy, but not restricted to interaction attributes, as they consider many other features, such as the curriculum design, productivity, management, and so on. However, they do not validate the utility of the ontology.

Considering all the previous works, and to the best of the authors’ knowledge, our work presented in this paper is the first that analyses the power of ontologies as a resource that makes the portability of EDM models easier and, in particular, it is also the only one for that purpose which is based on data from the students’ interactions with the LMS. Furthermore, it is the first research that depicts a comparative study against a previous non-ontological similar approach, with the purpose of demonstrating the performance improvement obtained when using ontologies. Both of these innovative aspects are the core contributions of this paper.

Materials and methods

In this section, we describe both the data used and the preprocessing tasks we applied to them in order to transform the raw data gathered from the Moodle logs to the high-level attributes of the proposed ontology. We also describe the experimentation that we carried out in order to address our research question.

Data and preprocessing

We have used the log data of 1840 Cordoba University students from 16 different courses taught by the Computer Science Department. Table 1 summarises these courses. For each course, it shows the subject or name of the course (Subject), our own identification Code, name of the Degree, Year in the degree/curriculum, number of students (#Users), and the level of Moodle Usage (Low, Medium or High). To accomplish the ethical and privacy issues about using these data, we have used informed consent with all the instructors and we have also anonymized all information about students (Pardo & Siemens, 2014).

Table 1 Information of all subjects
Full size table

We divided or grouped our 16 different courses (see Table 1) into three usage levels of Moodle activities in courses (see Table 2). Moodle provides us resources (text and web page, link to files or websites, and label) and different types of activities (assignments, chat, choice, database, forum, glossary, lesson, quiz, survey, wiki, workshop, etc.). We have defined three levels of usage by the number of activities used in the course:

  • Low level The course only has one or no activity.

  • Medium level The course has two different types of activities.

  • High level The course has three or more different types of activities.

Table 2 List of groups by Moodle usage
Full size table

Moodle provides a wide range of activities such as Assignments, Databases, Chats, Choice, Questionnaires, Quiz, Surveys, Forums, Glossaries, Lessons, SCORM packages, Workshops, Wikis, etc.). The most frequent activities in our courses are Assignments, Forums, and Quizzes. So, normally low-level courses only use one of these three activities, medium level two of them, and high level three or more activities. Table 2 shows the number of courses in each group grouped by usage level.

We also propose our ontology for defining 5 high-level attributes starting from 58 low-level attributes or actions provided by Moodle logs (see Table 3).

Table 3 Ontology and Moodle low-level actions of each category
Full size table

As depicted in Table 3, our ontology generalizes the 58 raw/low-level events provided by the Moodle logs into only five attributes or high-level categories. The first category references all the actions about consulting resources (LEARNING/READING/VIEWING), the second groups the students’ communication events (COMMUNICATING), the third deals with the students’ work (WORKING/DOING), the fourth is about students’ evaluation (EXAMINING/EVALUATING) and the last is about the students’ general ENGAGEMENT in the course. The first four attributes of our ontology are a number (from 0 to 100) that is the percentage of events of each type that each student has done in Moodle. The last attribute is the most general and is also a number (between 0 and 100) obtained from the total number of interactions/events and the number of days connected.

Finally, we have created two different datasets or data files: one with the original previously-described numerical data, and the other discretizing the attributes in two labels (HIGH and LOW) by using the equal width discretization method.

In both cases, we added a new attribute or class to predict at the end of our 5 attributes. This class is the final mark obtained by the students in the course, which is the value to predict in a classification task. The final mark (value between 0 and 10) has been discretized into two values or labels: FAIL if the student’s final mark is lower than 5 or PASS if the students’ final mark is higher than 5.

Methodology for experimentation

The methodology used in our experimentation consisted of these steps (see Fig. 1):

  • Firstly, we downloaded and preprocessed the Moodle log in order to obtain both the numerical and discretized datasets for each course. We used a specific Java tool that we developed for doing this specific transformation task (López-Zambrano et al., 2020).

  • Secondly, we executed the well-known J48 classification algorithm provided by the WEKA data mining environment for each one of the previous numerical and categorical datasets of 16 subjects or different courses. In this step, we obtained one prediction model for each course.

  • Then, we grouped our 16 subjects/courses into 3 groups depending on their level of usage of Moodle activities (see Table 1).

  • Next, we repeated the next two actions. We selected each prediction model obtained in one course one by one and we applied it to testing the datasets of all the other courses in the same group. We repeated this process with all the models and with all the datasets for each group.

  • Finally, we obtained the values of the two evaluation metrics that we used (the area under the ROC Curve and AUC loss) when applying the prediction model for one course/subject over the other datasets in the same group. And we compared the results obtained when using the original raw low-level

Fig. 1
figure 1

Methodology used in our experimentation

Full size image

Results

The results of these three groups are set out below (summarised in Table 3). Two experiments were conducted for each group, applying the J48 algorithm with balanced numerical and discretized datasets. These experiments consisted of having a first set of experiments for which high-level datasets were constructed (ontology) and a second experiment with datasets built with low-level attributes.

For each experiment (within the same group), we conducted an analysis of the best AUC obtained and the lowest error rate, or loss of portability, of the model. Thus, the results consist of two tables. At the top, a matrix is shown with the results of the AUC metric, obtained from the list of the general model for each subject (rows), compared to the average AUC for the individual datasets from each period for a subject (columns). The values of the main diagonal represent the testing of the general model for subjects over their own datasets, where this value is the reference AUC value (highest value), with regard to the AUCs from the other subjects. The second matrix (bottom) displays the difference between the highest AUC (reference) by row, with regard to each individual AUC. These values tell us how much precision is lost in the AUC when this model is tested with other subjects (portability), aiming to highlight the lowest values, as they indicate the lowest error rate or loss in the process of model portability or transferability.

Group of courses with high-level usage

For the high-level group, we can see in Table 4 that of the two tests, the best general results (averages) are in the datasets with ontology, revealing that the AUC average for numerical datasets is 0.62 and the average for discretized datasets is 0.61, higher than their equivalents in the tests without ontology. While there is only a small difference, the loss rate or difference in transferability does denote a greater difference, and within the same test group, the difference between numerical and discretized datasets is highly significant, where the tests with discretized data are much better.

Table 4 AUC results and loss of transferability (difference) with J48—high-level group
Full size table

If we focus on the tests with the best results, we can see that the best value for the AUC metric (0.675) is in the ICS2 subject obtained with discretized data and tested with the subject HCI. This is not concordant with the general average of the AUCs, whose highest value is for the numerical sets (0.62), with a tiny difference of one one-hundredth. However, it is concordant with the fact that the best rate of loss or difference is with the discretized data (0.10). We can also see that the generalized model obtained with the aforesaid subject (ICS2) has very good results, which is proven in the general averages (row) in both tests (with and without ontology).

With regard to the model that obtained the best average in the precision loss rate, we can see in Fig. 2, the decision tree, defining the attribute COMMUNICATING (from the five ontology attributes—Table 1) as the attribute with the highest increase in information, which would define the prediction for a student passing the course.

Fig. 2
figure 2

The best model for the high-level group with discretized dataset—subject ICS2

Full size image

Group of courses with medium-level usage

For the medium-level group, we can see in Table 5 that of the two tests, the best general results (averages) are in the dataset tests with ontology, revealing that the AUC average for numerical datasets is 0.60 and the average for discretized datasets is 0.59, higher than their equivalents in the tests without ontology. There is a small difference, although the loss rate or difference in transferability does denote a greater difference, and within the same test group, the difference between numerical and discretized datasets is highly significant, where the tests with discretized data are much better.

Table 5 AUC results and loss of transferability (difference) with J48—medium-level group
Full size table

If we focus on the tests with the best results, we see that the best value for the AUC metric (0.718) is for the subject SDC, obtained with numerical data and tested with the subject RE, which is concordant with the general AUC average, whose highest value is in the numerical tests (0.60), with is a small difference of one one-hundredth. However, it is not concordant with the fact that the best loss or difference rate is for discretized data (0.18). We also see that in the generalized model within the tests with discretized data, the subject PM1 has a good result in the general average for the loss rate (row) in the tests without ontology, although in the tests with ontology (employing a generalized model of high-level attributes), there are also good results in the IP1 and INS subjects, which share the same value of 0.13, six one-hundredths more, but still within the ideal value for good transferability of the model.

With regard to the subjects with the best average loss rate, Fig. 3 shows that the decision tree defines the attributes LEARNING/READING/VIEWING and COMMUNICATING (from the five ontology results—Table 1) as the attributes with the greatest gain in information, defining that if there is a high level of LEARNING/READING/VIEWING, the student will pass or, conversely, if it is low, but with a high level of interaction in COMMUNICATING, the student will also pass.

Fig. 3
figure 3

Best model for the medium-level group with discretized dataset—subject IP1

Full size image

Concerning the decision tree shown in Fig. 4, it defines the attributes LEARNING/READING/VIEWING, COMMUNICATING, WORKING/DOING and EVALUATING/EXAMINING (from the five ontology attributes—Table 1) as the attributes with the greatest increase in information, once again defining that if there is a high level of LEARNING/READING/VIEWING, the student will pass or, conversely, if it is low, but with a high level of interaction in COMMUNICATING, the student will also pass. If the COMMUNICATING level is low, but the level of WORKING/DOING is high, then the student would pass, but if it is not high, then the student will only pass if the EVALUATING/EXAMINING level is high.

Fig. 4
figure 4

Best model for the medium-level group with discretized dataset—subject InS

Full size image

Group of courses with low-level usage

For the low-level group, we can see in Table 6 that of the two tests, the best general results (averages) are in the dataset tests with ontology, revealing that the AUC averages for numerical datasets is 0.63 and the average for discretized datasets is 0.61, higher than their equivalents in the tests without ontology. There is a small difference, although the loss rate or difference in transferability does denote a greater difference, and within the same test group, the difference between numerical and discretized datasets is highly significant, where the tests with discretized data are much better.

Table 6 AUC results and loss of transferability (difference) with J48—low-level group
Full size table

If we now focus on the tests with the best results, we can see that the best value for the AUC metric (0.683) is in the ICS1 subject obtained with discretized data and tested with the subject ICS4. This is not concordant with the general average of the AUCs whose highest value is for the numerical sets (0.63), with a small difference of two one-hundredths. However, it is concordant with the fact that the best rate of loss or difference is with the discretized data (0.13). We can also observe that the generalized model obtained with the aforesaid subject (ICS1) has very good results, which is proven in the general averages (row) for the matrix of discretized data with ontology. The value is 0.07, which is below the ideal for determining a good transfer of the model, in this case for a general model with high-level attributes.

With regard to the subject with the best average loss rate, we can see in Fig. 5, the decision tree, defining the attribute COMMUNICATING (from the five attributes of ontology—Table 1) as the attribute with the highest increase in information, which would define the predictability for a student passing the course.

Fig. 5
figure 5

Best model for the low-level group with discretized dataset—subject ICS1

Full size image

Conclusions

This paper aims to improve the portability or transferability of predictive models of students’ performance by using an ontology that uses a taxonomy of actions on students’ interactions with the Moodle learning management system. We compare the results of this new proposed approach against our previous results when we used low-level raw attributes directly obtained from Moodle logs. The results obtained show that the use of the proposed ontology significantly improves the portability of the models in terms of their predictive accuracy. So, the answer to our initial research question is yes, the ontological models obtained in one source course can be applied to other different target courses with similar usage levels without losing prediction accuracy.

One of the limitations of this work is the specific attributes/variables used in our proposed ontology.

For example, it is also important to discuss if the “number of total interactions” are truly showing engagement when learning using LMS. The number of actions includes the behavior of supposed relevant activity in the LMS and were are assuming that all of these actions could indicate that the student is properly involved in his learning process. As traditionally happens with study time, however, this variable by itself is very tricky. It may seem that the more time those students spend studying, the better grades they should receive, but it is not that simple; it mainly depends on the quality of the study time, and something similar could be occurring with the relevant actions; more activity in the LMS does not assure better results (Cerezo et al., 2016).

Regarding the application of the results obtained in this work and the potential for using them within other domains; it is important to notice that currently there is an increasing interest in the generalization and portability of prediction models and specifically with Moodle LMS (Monllao-Olive et al., 2019). In this line, our proposal can be applied not only to Learning Management Systems as Moodle but also to other different domains or data sources such as Intelligent Tutoring Systems (ITSs), Massive Online Open Courses (MOOCs), Traditional face-to-face educational environments, Blended Learning and Multimodal Learning environments, and so on.

Finally, as a future study, we are currently working on:

  • Using a higher number of courses with much more data/students from different areas/domains, not only engineering and computer science, but also fields such as science, biology, medicine, philosophy, and literature, in order to generalize the good results that we obtained in this study.

  • Discovering predictive models that can be portable/transferable as soon as possible in the early stages of the course. This means we would not have to wait until the end of the course to have all Moodle usage data available, and the obtained models could be used as general early warning prediction models for different similar courses (Romero & Ventura, 2019).