A Novel Integrated PIPRECIA–Interval-Valued Triangular Fuzzy ARAS Model: E-Learning Course Selection

Abstract

The development of information and communication technologies has revolutionized and changed the way we do business in various areas. The field of education did not remain immune to the mentioned changes; there was a gradual integration of the educational process and the mentioned technologies. As a result, platforms for distance learning, as well as the organization of e-learning courses of various types, have been developed. The rapid development of e-learning courses has led to the problem of e-learning course selection and evaluation. The problem of the e-learning course selection can be successfully solved by using multiple-criteria decision-making (MCDM) methods. Therefore, the aim of the paper is to propose an integrated approach based on the MCDM methods and symmetry principles for e-learning course selection. The pivot pairwise relative criteria importance assessment (PIPRECIA) method is used for determining the weights of criteria, and the interval-valued triangular fuzzy additive ratio assessment (ARAS) method is used for the ranking of alternatives i.e., e-learning courses. The suitability of the proposed integrated model is demonstrated through a numerical case study.

1. Introduction

The role of information and communication technologies (ICT) has become very important in all aspects of life and business. These technologies have a very significant role in information elaboration and its transformation into knowledge, which is the main condition for someone to become an efficient part of the information society [1]. The information has become accessible and transferable from anywhere in the world. Also, education and its availability have an important role in the modern world and therefore, ICT has become an integral part of people’s lives in all aspects [2]. The rapid and dynamic development of information and communication technologies has led to significant changes in people’s personal, social, and work lives. Currently, people live in an information society that relies on the use of information and communication technologies, which is also a society of knowledge, intangible capital and learning, in which progress is based on knowledge and creativity [3]. The education did not remain immune to the spreading of the mentioned technologies and therefore, these technologies had integrated into the learning process, as well.

Education is the constitutive element of the knowledge society and global economy of knowledge if it is structured in line with information and communication technologies. Furthermore, the integration of these technologies could make education more available, but also could change the cultural context of education, as well as the language of learning [4]. These impacts change the way of learning because the students do not receive the knowledge passively as they have previously but they are actively involved in the learning process. Wang and Woo [5] state that the integration of information and communication technologies is often seen as a process of deploying any ICT (including resources on the Internet, multimedia programs, learning objects, and other tools) with the aim of improving student learning. Although the primary function of ICT use is not only in the field of education, it can be said that information and communication technologies are a tool for achieving learning goals [6].

In this competitive world, the future of the education sector is based on knowledge of ICT. The integration of ICT into education through e-learning courses enables active learning, discussion, sharing of ideas, immediate feedback, and easy access to digital content [7]. When it comes to technology integration into education, three main components should be taken into account: content, pedagogy, and technology [8]. The integration of ICT in educational processes is often defined as the process of applying any ICT (including Internet resources, multimedia programs, learning subjects, and other tools) with the aim to improve student learning and achieve desired outcomes [5,9].

Effective integration of ICT and learning processes through e-learning courses has great potential to involve and engage students on a larger scale. For example, the use of multimedia for presenting authentic and poorly structured problems in problem-based learning can motivate and engage students and thus help them to develop problem-solving skills [10]. An and Reigeluth [11] emphasize that groups can have “different learning needs based on their interests and problem-solving plans, even though if they were working on the same problem”. E-learning courses can support different types of interaction: student–content, student–student, student–lecturer, and student–interface [12]. These types of interactions make the learning process more interactive, and students more active and engaged.

Studies have shown that online education can be just as effective as the traditional way of education, and in some cases, it has even been shown to be better [13,14]. According to Stavredes and Herder [15], creating an effective e-learning course begins with understanding the needs of future students. Therefore, it is necessary to identify the needs and requirements of future users in order to create such a course that will successfully meet the stated needs. Also, it is important to point out that the e-learning courses have enabled the acquisition of quality education to those participants for whom education would not be available in other circumstances.

As stated, integration of the learning process and information and communication technologies has led to the development of the e-learning courses. In that way, the educational resources are available to the individuals who, in different conditions, are able to attend the lectures. The very important question is the design of the course so that it is understandable and easy to use for different kinds of students. The creators of the e-learning courses should pay attention to the features that enable the smooth operation of the course, which is very important. The appearance of the e-learning courses initiated the need to find suitable methods for their evaluation and selection.

Multiple-criteria decision-making (MCDM) is often associated with the selection of an alternative from a set of alternatives, but it can also be used for ranking alternatives [16,17]. Until now, a number of MCDM methods have been proposed, such as: SAW [18], AHP [19], TOPSIS [20], PROMETHEE [21], ELECTRE [22], VIKOR [23], and so forth.

Notable progress in solving real-world complex decision-making problems appeared after Zadeh [24] and his introduction to the fuzzy sets theory. As a part of the fuzzy set theory, fuzzy numbers that are usually based on triangular or trapezoidal shapes, which are much more adequate for modeling and solving a number of complex decision-making problems were introduced.

Based on the previously mentioned fuzzy sets theory, somewhat later Bellman and Zadeh [25] introduced the fuzzy MCDM-based methodology, which was later widely accepted in the scientific community as well as being used to solve many decision-making problems. With the aim to solve a variety of complex MCDM problems, some extensions to the fuzzy set theory have been proposed, such as intuitionistic fuzzy sets [26], interval-valued fuzzy sets [27], bipolar fuzzy sets [28], and so on.

The interval-valued fuzzy numbers (IVFNs), as a particular form of fuzzy numbers, provide much more possibilities for solving the real-world MCDM problems. Hence, some of the prominent multiple-criteria decision-making methods have proper extensions based on IVFNs. It is worth mentioning some of them: ELECTRE [29], VIKOR [30,31], MULTIMOORA [32], TOPSIS [33,34,35], and so forth.

The additive ratio assessment (ARAS) method was developed by Zavadskas and Turskis [36]. The ARAS method has been applied to solve various decision-making problems. So, for example, Zavadskas and Turskis [36] have applied the ARAS method to evaluate the microclimate in office rooms, Zavadskas et al. [37] have applied the ARAS method to select the most appropriate foundation installment alternative, Karabašević et al. [38] have applied the ARAS method for personnel selection. Besides that, the ARAS method has also been applied for the ranking of companies according to the CSR indicators [39], selection of the software testing method [40], mineral prospectivity mapping [41], reduction of greenhouse gas emission [42], and so on.

In order to extend the applicability of the ARAS method and to enable the use of grey and fuzzy numbers, Turskis and Zavadskas [43] proposed a proper fuzzy extension (ARAS-F) and a grey extension (ARAS-G) [44]. It is also important to note that on the basis of the ARAS method, some other approaches are proposed. In this context, it is worth mentioning the ARCAS approach proposed by Stanujkic et al. [45] that is based on ordinary SWARA and ARAS methods and is adapted for negotiations.

Based on all the above, the main aim of this paper is to propose an integrated approach based on the MCDM methods for the selection of the e-learning course. The proposed approach is based on the use of the interval-valued triangular fuzzy (IVTFN) ARAS method for the ranking of alternatives and the PIPRECIA method for the determining weights of the criteria.

Therefore, the rest of this paper is organized as follows. Section 2 demonstrates the proposed methodology, based on the PIPRECIA and extended ARAS method. In order to highlight the proposed MCDM methodology, in Section 3 a case study of e-courses evaluation is considered. The discussions and conclusions are given at the end of the manuscript.

2. Methods

The proposed methodology for the evaluation of the e-courses is based on the PIPRECIA method and interval-valued triangular fuzzy ARAS method. The PIPRECIA method is used for defining weights of the criteria whereas the interval-valued triangular fuzzy ARAS method is used for the ranking of alternatives.

2.1. PIPRECIA Method

The pivot pairwise relative criteria importance assessment (PIPRECIA) method for determining the weights of criteria was proposed by Stanujkic et al. [46], as one of the extensions of the step-wise weight assessment ratio analysis (SWARA) method. Although it is a relatively new method, so far, the PIPRECIA method is applied for solving problems of personnel selection [47], assessment of ICT [48,49], supplier selection [50], mining method selection [51], and so forth.

The computational procedure of the PIPRECIA method can be demonstrated through the following steps [46]:

Step 1. Determination of the set of the evaluation criteria.

Step 2. Starting from the second criterion, it is necessary to determine the relative importance s_j of the criterion j in relation to the previous (j − 1) criterion:

$$s_j=\{\begin{array}{rr}\hfill >1& \hfill when significance of C_j\succ C_{j-1}\\ \hfill 1& \hfill when significanse of C_j=C_{j-1}\\ \hfill <1& \hfill when significance of C_j\prec C_{j-1}\end{array}.$$

(1)

Step 3. Calculation of the coefficient k_j as follows:

$$k_j=\{\begin{array}{cc}1& j=1\\ 2-s_j& j>1\end{array}$$

(2)

Step 4. Calculation of the recalculated weight q_j as follows:

$$q_j=\{\begin{array}{cc}1& j=1\\ \frac{q_{j-1}}{k_j}& j>1\end{array}.$$

(3)

Step 5. Calculation of the relative weights of the evaluation criteria as follows:

$$w_j=\frac{q_j}{{\displaystyle \sum _{k=1}^n{q}_k}}$$

(4)

where w_j represents the relative weight of the criterion j.

2.2. An Extension of the ARAS Method Based on the Use of Interval-Valued Fuzzy Numbers

Based on Stanujkic et al. [52], the computational procedure for selecting the most acceptable alternative by applying the IVTFN-ARAS method that includes only beneficial criteria could be demonstrated through the following steps:

Step 1. Determination of the optimal performance rating for each criterion.

$${\tilde{x}}_{0j}=[(l_{0j},l_{0j}^{\prime }),m_{0j},(u_{0j}^{\prime },u_{0j})],$$

(5)

with

$$l_{0j}=\underset{i}{\max } l_{ij},$$

(6)

$$l_{0j}^{\prime }=\underset{i}{\max } l_{ij}^{\prime },$$

(7)

$$m_{0j}=\underset{i}{\max } m_{ij},$$

(8)

$$u_{0j}^{\prime }=\underset{i}{\max } u_{ij}^{\prime }$$

(9)

$$u_{0j}=\underset{i}{\max } u_{ij}$$

(10)

where ${\tilde{x}}_{0j}$ represents the interval-valued fuzzy optimal performance rating of criterion j.

Step 2. Calculation of the normalized decision matrix.

$${\tilde{r}}_{ij}=\left[\left(\frac{a_{ij}}{c_j^{+}},\frac{a_{ij}^{\prime }}{c_j^{+}}\right),\frac{b_{ij}}{c_j^{+}},\left(\frac{c_{ij}^{\prime }}{c_j^{+}},\frac{c_{ij}}{c_j^{+}}\right)\right],$$

(11)

where ${\tilde{r}}_{ij}$ represents the normalized interval-valued fuzzy performance rating of alternative i in relation to the criterion j, $c_j^{+}={\displaystyle {\sum }_{i=0}^m{c}_{ij}}$.

Step 3. Calculation of the weighted interval-valued normalized fuzzy decision matrix.

$${\tilde{v}}_{ij}=w_j·{\tilde{r}}_{ij}$$

(12)

where ${\tilde{v}}_{ij}$ represents the weighted normalized interval-valued fuzzy performance rating of alternative i in relation to the criterion j.

Step 4. Calculation of the overall interval-valued fuzzy performance ratings.

$${\tilde{S}}_i={\displaystyle \sum _{j=1}^n{w}_j{\tilde{r}}_{ij}},$$

(13)

where ${\tilde{S}}_i$ represent overall interval-valued fuzzy performance rating of alternative i.

Step 5. Calculation of the degree of utility, for each alternative. As a result of performing the previous steps, the obtained overall performance ratings are IVFNs. Therefore, overall performance ratings have to be defuzzified before the calculation of the overall degree of utility. In this way, the same equation as in the ordinary ARAS method is used to determine the overall degree of utility.

Step 6. Ranking of alternative selections by the most efficient. This step is the same as in the original ARAS method.

3. A Numerical Case Study

This section describes the process of the selection of an e-learning course based on the opinions of twenty-four respondents by using the PIPRECIA method and the IVTFN-ARAS method. Four e-learning courses in the field of programming are evaluated, designated as: A₁—Cubes (www.cubes.edu.rs); A₂—ITAcademy (www.it-akademija.com); A₃—Link-eLearning (www.link-elearning.com), and A₄—Ok School (www.ok-school.com). The criteria used for the evaluation of the e-learning courses are obtained based on the analysis of the relevant literature [53,54,55], and their weights are determined by using the PIPRECIA method. Evaluation criteria and their corresponding weights that represent the attitudes of the group are shown in

Table 1. The group evaluation criteria and their weights.

Criteria		wj
C1	Level of content	0.14
C2	Presentation method	0.14
C3	Teaching method	0.13
C4	e-learning environment	0.14
C5	Learning materials	0.15
C6	Quality of multimedia content	0.14
C7	Group work and interactivity	0.15

.

At the beginning of the evaluation, all twenty-four respondents evaluated alternatives using the five-point Likert scale. The Likert scale was chosen because it is easy to use and its usage is understandable for respondents. Ratings obtained from three randomly selected respondents are shown in

Table 2. The ratings obtained from the fist of twenty-four respondents.

Criteria	C1	C2	C3	C4	C5	C6	C7
A1	4	3	3	4	2	4	5
A2	3	5	2	4	4	4	4
A3	5	5	4	5	3	3	2
A4	4	5	5	4	4	4	4

,

Table 3. The ratings obtained from the second of twenty-four respondents.

Criteria	C1	C2	C3	C4	C5	C6	C7
A1	2	3	5	5	2	3	2
A2	4	5	4	5	4	4	5
A3	5	4	4	5	5	5	3
A4	3	5	3	4	4	5	4

and

Table 4. The ratings obtained from the third of twenty-four respondents.

Criteria	C1	C2	C3	C4	C5	C6	C7
A1	3	3	3	4	5	4	4
A2	2	4	4	3	2	5	5
A3	5	4	4	3	4	5	3
A4	5	5	5	3	3	4	4

and Figure 1, Figure 2 and Figure 3.

Subsequently, in the next step, a group decision matrix was formed. The elements of this matrix, shown in

Table 5. IVTF group decision-making matrix.

	C1	C2	C3	C4	C4	C6	C7
A1	[(1, 2.5), 3.25, (4.3, 5)]	[(2, 2.67), 3.5, (4.33, 5)]	[(2, 2.75), 3.63, (4.5, 5)]	[(1, 2.11), 3.46, (4.27, 5)]	[(1, 2.4), 3.13, (4.33, 5)]	[(1, 2), 3.04, (4.27, 5)]	[(1, 2.33), 3.13, (4.44, 5)]
A2	[(2, 2.69), 3.29, (4.5, 5)]	[(2, 2.92), 3.71, (4.64, 5)]	[(2, 2.4), 3.5, (4.29, 5)]	[(2, 2.44), 3.54, (4.2, 5)]	[(1, 2.44), 3.46, (4.07, 5)]	[(2, 2.17), 3.71, (4.22, 5)]	[(1, 2.44), 3.58, (4.27, 5)]
A3	[(1, 2.8), 3.92, (4.71, 5)]	[(1, 2.64), 3.71, (4.62, 5)]	[(1, 2.31), 3.17, (4.18, 5)]	[(1, 2.3), 3.79, (4.86, 5)]	[(1, 2.83), 3.54, (4.25, 5)]	[(1, 2.4), 3.63, (4.5, 5)]	[(2, 2.53), 3.13, (4.57, 5)]
A4	[(2, 2.58), 3.46, (4.33, 5)]	[(1, 3), 4, (4.67, 5)]	[(2, 2.58), 3.63, (4.67, 5)]	[(1, 2.3), 3.38, (4.14, 5)]	[(1, 2.5), 3.79, (4.44, 5)]	[(1, 2.15), 3.13, (4.27, 5)]	[(2, 2.5), 3.38, (4.25, 5)]

, are IVTFNs formed by the transformation of crisp ratings into IVTFNs, as is explained in Stanujkic [52].

Based on the data from Table 5, the optimal performance ratings are determined by using Equation (5). The obtained optimal performance ratings are shown in

Table 6. The optimal IVTF performance ratings.

	C1	C2	C3	C4	C4	C6	C7
A0	[(2, 2.8), 3.92, (4.71, 5)]	[(2, 3), 4, (4.67, 5)]	[(2, 2.75), 3.63, (4.67, 5)]	[(2, 2.44), 3.79, (4.86, 5)]	[(1, 2.83), 3.79, (4.44, 5)]	[(2, 2.4), 3.71, (4.5, 5)]	[(2, 2.53), 3.58, (4.57, 5)]

.

In the next two steps, normalized and weighted normalized decision-making matrices are calculated using Equations (11) and (12). The normalized and weighted normalized decision-making matrices are shown in

Table 7. The normalized IVTF performance rating.

	C1	C2	C3	C4	C4	C6	C7
A0	[(0.4, 0.56), 0.78, (0.94, 1)]	[(0.4, 0.6), 0.8, (0.93, 1)]	[(0.4, 0.55), 0.73, (0.93, 1)]	[(0.4, 0.49), 0.76, (0.97, 1)]	[(0.2, 0.57), 0.76, (0.89, 1)]	[(0.4, 0.48), 0.74, (0.9, 1)]	[(0.4, 0.51), 0.72, (0.91, 1)]
A1	[(0.2, 0.5), 0.65, (0.86, 1)]	[(0.4, 0.53), 0.7, (0.87, 1)]	[(0.4, 0.55), 0.73, (0.9, 1)]	[(0.2, 0.42), 0.69, (0.85, 1)]	[(0.2, 0.48), 0.63, (0.87, 1)]	[(0.2, 0.4), 0.61, (0.85, 1)]	[(0.2, 0.47), 0.63, (0.89, 1)]
A2	[(0.4, 0.54), 0.66, (0.9, 1)]	[(0.4, 0.58), 0.74, (0.93, 1)]	[(0.4, 0.48), 0.7, (0.86, 1)]	[(0.4, 0.49), 0.71, (0.84, 1)]	[(0.2, 0.49), 0.69, (0.81, 1)]	[(0.4, 0.43), 0.74, (0.84, 1)]	[(0.2, 0.49), 0.72, (0.85, 1)]
A3	[(0.2, 0.56), 0.78, (0.94, 1)]	[(0.2, 0.53), 0.74, (0.92, 1)]	[(0.2, 0.46), 0.63, (0.84, 1)]	[(0.2, 0.46), 0.76, (0.97, 1)]	[(0.2, 0.57), 0.71, (0.85, 1)]	[(0.2, 0.48), 0.73, (0.9, 1)]	[(0.4, 0.51), 0.63, (0.91, 1)]
A4	[(0.4, 0.52), 0.69, (0.87, 1)]	[(0.2, 0.6), 0.8, (0.93, 1)]	[(0.4, 0.52), 0.73, (0.93, 1)]	[(0.2, 0.46), 0.68, (0.83, 1)]	[(0.2, 0.5), 0.76, (0.89, 1)]	[(0.2, 0.43), 0.63, (0.85, 1)]	[(0.4, 0.5), 0.68, (0.85, 1)]

and

Table 8. The weighted IVTF performance ratings.

	C1	C2	C3	C4	C4	C6	C7
w	0.14	0.14	0.13	0.14	0.15	0.14	0.15
A0	[(0.06, 0.08), 0.11, (0.14, 0.14)]	[(0.06, 0.08), 0.11, (0.13, 0.14)]	[(0.05, 0.07), 0.1, (0.13, 0.13)]	[(0.06, 0.07), 0.11, (0.14, 0.14)]	[(0.03, 0.09), 0.11, (0.13, 0.15)]	[(0.06, 0.07), 0.11, (0.13, 0.14)]	[(0.06, 0.08), 0.11, (0.14, 0.15)]
A1	[(0.03, 0.07), 0.09, (0.12, 0.14)]	[(0.06, 0.07), 0.1, (0.12, 0.14)]	[(0.05, 0.07), 0.1, (0.12, 0.13)]	[(0.03, 0.06), 0.1, (0.12, 0.14)]	[(0.03, 0.07), 0.09, (0.13, 0.15)]	[(0.03, 0.06), 0.09, (0.12, 0.14)]	[(0.03, 0.07), 0.09, (0.13, 0.15)]
A2	[(0.06, 0.08), 0.1, (0.13, 0.14)]	[(0.06, 0.08), 0.1, (0.13, 0.14)]	[(0.05, 0.06), 0.09, (0.12, 0.13)]	[(0.06, 0.07), 0.1, (0.12, 0.14)]	[(0.03, 0.07), 0.1, (0.12, 0.15)]	[(0.06, 0.06), 0.11, (0.12, 0.14)]	[(0.03, 0.07), 0.11, (0.13, 0.15)]
A3	[(0.03, 0.08), 0.11, (0.14, 0.14)]	[(0.03, 0.07), 0.1, (0.13, 0.14)]	[(0.03, 0.06), 0.09, (0.11, 0.13)]	[(0.03, 0.07), 0.11, (0.14, 0.14)]	[(0.03, 0.09), 0.11, (0.13, 0.15)]	[(0.03, 0.07), 0.1, (0.13, 0.14)]	[(0.06, 0.08), 0.09, (0.14, 0.15)]
A4	[(0.06, 0.07), 0.1, (0.13, 0.14)]	[(0.03, 0.08), 0.11, (0.13, 0.14)]	[(0.05, 0.07), 0.1, (0.13, 0.13)]	[(0.03, 0.07), 0.1, (0.12, 0.14)]	[(0.03, 0.08), 0.11, (0.13, 0.15)]	[(0.03, 0.06), 0.09, (0.12, 0.14)]	[(0.06, 0.07), 0.1, (0.13, 0.15)]

.

Finally, the overall interval-valued triangular fuzzy performance ratings, obtained by using Equation (13), are shown in

Table 9. The overall IVTF performance ratings.

A0	[(0.07, 0.11), 0.15, (0.19, 0.2)]
A1	[(0.05, 0.1), 0.13, (0.17, 0.2)]
A2	[(0.07, 0.1), 0.14, (0.17, 0.2)]
A3	[(0.05, 0.1), 0.14, (0.18, 0.2)]
A4	[(0.06, 0.1), 0.14, (0.18, 0.2)]

.

In order to determine the quality of the e-courses, these values must be defuzzified using some of the well-known procedures [52].

Results obtained using the simplest of all of the considered defuzzification procedures are shown in

Table 10. The degree of utility and ranking order of analyzed e-courses.

Alternatives	BNP	Qi	Rank
A0	0.143
A1	0.130	0.910	4
A2	0.136	0.951	1
A3	0.134	0.936	3
A4	0.135	0.941	2

. The relative quality, i.e., the degree of utility, of analyzed e-courses as well as their ranking orders, are also shown in Table 10.

As can be seen from Table 10, the best alternative, i.e., e-course, is the alternative designated as A₂.

By varying the coefficient λ, greater importance can be given to l and u in relation to l’ and u’, and vice versa. The results obtained using Equation (16) for some characteristic values of the coefficient λ are shown in

Table 11. The degree of utility and ranking order of analyzed e-course for some characteristic values of λ.

		λ = 0			λ = 0.5			λ = 1
Alternatives	Si	Qi	Rank	Si	Qi	Rank	Si	Qi	Rank
A0	0.42			0.43			0.44
A1	0.39	2.72	4	0.39	2.73	4	0.39	2.73	4
A2	0.41	2.87	1	0.41	2.87	1	0.41	2.87	1
A3	0.40	2.81	3	0.40	2.81	3	0.40	2.81	3
A4	0.41	2.83	2	0.41	2.83	2	0.41	2.83	2

.

In this case, the alternative denoted as A₂ remains the best alternative i.e., an e-learning course in all cases. This indicates the stability of the chosen e-learning course.

However, in many cases variation of the coefficient lambda may have an impact on the ranking order of the considered alternatives, and this approach may be useful to analyze different scenarios such as pessimistic, realistic, and optimistic.

4. Discussion and Conclusions

The main aim of the paper is to introduce a methodology suitable for e-learning course evaluation and selection. The emergence of e-courses as the modern way of learning has provoked the need for finding the methods ideal for their assessment. In this paper, the methodology based on the PIPRECIA and the interval-valued triangular fuzzy ARAS methods is proposed. The applicability of the proposed methodology is presented through a numerical case study. When defining criteria, special attention was dedicated to the issues of organization and teaching. So, in this case, the attention was not directed on the technical and informational performance of an e-learning course, but towards the quality of the offered content and how the teaching process was implemented.

Based on the data obtained from the respondents, the PIPRECIA method was applied, and the weights of the criteria for each of the twenty-four respondents were obtained, as is shown in Table 1. The obtained results show that the weights of the criteria are approximate, which impose the fact that the given features are nearly equally important to all respondents. This is entirely understandable because the e-learning course should satisfy all the requirements and, in that way, offer the quality of “service” to the users.

The reason for applying the PIPRECIA method for the determination of the weights lies in its simplicity and suitability for use in cases where a large number of decision-makers are involved in the evaluation process. The advantage of the PIPRECIA method over the well-known and widely used AHP method is reflected in a more straightforward computational procedure that does not diminish the reliability and relevance of the results obtained. Also, when interviewing respondents who are not familiar with the MCDM methods, the process of evaluating weights by using the PIPRECIA method is far more understandable to respondents, than is the case with the AHP method. If the PIPRECIA method is compared with the SWARA method (on which the PIPRECIA method was developed), it can be concluded that the PIPRECIA method has certain advantages over it. Namely, the SWARA method requires that the evaluation criteria should be sorted according to their intended significance, which complicates its application in group decision cases. Many complex decision-making problems require the participation of a group of respondents. In such cases, the individual attitudes of the respondents have to be transformed into group attitudes, with an as small as possible loss of information. In order to take into account the uncertainty and imprecision of the data on which decision-making is very often based, the application of the interval-valued triangular fuzzy ARAS method is proposed. The approach in which individual ratings are transformed into interval-valued triangular fuzzy numbers can be very useful in this regard. The interval-valued triangular fuzzy ARAS method may use such information to rank alternatives and/or analyze different scenarios. Thus, by applying this method, decision-makers have been given the opportunity to express their optimistic, pessimistic, and realistic attitudes.

In this paper, the numerical case study of the e-learning course selection was examined. The reason for that relies on the increasing importance of this kind of learning. In order to create the high quality e-learning course, it is necessary to determine the pros and cons of the considered course and its position relative to the competition. In that way, the creators will know what aspects of the course should be improved and what are of satisfactory quality. The application of the proposed integrated approach has proven to be quite justified and appropriate in this case. The reason is that if the e-learning courses were evaluated based solely on the use of crisp numbers, the obtained results would not include uncertainty. This would result in a decision that would not be completely realistic and, ultimately, unreliable. The obtained results confirmed this point of view. To get the most reliable results and to make the best possible decisions, it is necessary to respect the risk and uncertainty to the maximum extent possible. So, based on the conducted numerical case study, the e-learning course designated as A₂ is the best in terms of evaluated criteria.

As the examination of the literature has shown, the authors used different approaches for e-course evaluations. Chao and Chen [56] examined which factors are crucial for the quality of the e-learning courses. They applied the consistent fuzzy preference relations (CFPR) with AHP methodology. They evaluated four groups of factors that are elaborated in a particular number of criteria. The final results showed that the most influential criteria are: the e-learning material, friendly user interface, using the web discussion zone, and distant learning without time and space. The main point of this paper is the quality of the content of the e-learning courses. The assessment of the evaluation criteria showed that e-learning material has the greatest influence together with the group work and interactivity, which is in line with the results obtained from the mentioned authors.

Garg and Jain [57] applied the combination of the methods for defining the best e-learning website. They divide the evaluation criteria into two groups called quality factors and e-learning specific factors. The second group of factors is pointed to the quality of e-learning content and their results showed that the most important criterion is the ease of learning community, which could be considered as a counterpart to the group work and interactivity presented here.

Besides the mentioned works, others present the utilization of different MCDM techniques for e-learning course evaluation and selection. For the resolving of the problem of evaluation of e-course quality, the authors have proposed the application of the proximity indexed value (PIV) model [58], fuzzy ANP [59], DANP and VIKOR [60], and so on. The conducted numerical case study presented in this paper, as well as the comparison with the results of the given authors, confirmed that the proposed approach is also very beneficial for e-learning course evaluation and selection because the obtained results clearly outlined the crucial characteristics and the position of the specific e-learning course comparing to the others.

Therefore, the proposed integrated PIPRECIA-IVTFN-ARAS model has proven to be useful and feasible, especially in circumstances where it is essential to make the most relevant and realistic decision possible. The proposed integrated model can be extended to other areas of the business as well.

As a direction for future research, a significantly larger sample could be used in order to obtain results from a macro point of view. Besides that, the learning outcomes of the learners that would study the ranked courses can be further investigated as well.