Introduction

Higher education institutions (HEIs) are reconsidering the way they deliver education to the public since openness in education has been introduced. However, although the concept of openness in higher education causes substantial attention for possible benefits, it is also a source of discussion (Kalz 2014). One of the motives feeding the debate surrounding openness in education is the concept itself, as it is not a unified entity. The term “open” often means different things to different individuals confirming an early statement of Hyland (1979) who described the field as “eclectic”. In addition, Bayne et al. (2015) argue that the “field has lacked coherent definitions of ‘open’, and too often tended towards optimism, advocacy, and conviction, rather than a critical understanding of what openness might mean for education”.

The confusion around the concept results from the fact that open education definitions and projects have been developed in different contexts, with differing priorities. The term can for instance describe the open publication, use and reuse of learning resources, it can describe specific types of teaching and learning practices, institutional practices, the use of technology in education, and principles and values at the core of educational activities (Sampson and Zervas 2014). This complexity also mirrors the historical development of the open education movement (Sabadie et al. 2014). Therefore, the term “open education” is regarded as an umbrella term (Weller 2015). However, in a research context, it is important to identify the many interpretation(s) and contexts of openness, especially since the underlying ideas behind these different interpretations and contexts can yield different results.

Although openness in higher education seems to be this granular concept, there are broad subcategories defined that distinguish two sides of a continuum of openness in education. Within the category of open educational resources (OER) the focus is on the development, use (and reuse) of open educational content. On the other hand, the open online education (OOE) movement is more focused on scalable delivery of open courses, connected pedagogies and open sharing of teaching practices, going beyond content provision (Kalz 2014). While the emphasis of the OER sub-community is on building more access to digital educational content and reuse and adaptation of this content, the OOE sub-community, to which the activities around Massive Open Online Courses (MOOCs) belong, is busy with educational approaches to realise online learning at scale. Both sides of the continuum are criticized: On the one hand the OER approach has been criticized regarding its lack of process-oriented ideas on education (Knox 2013a) and a general lack of ideas regarding teaching (Knox 2013b). In other words, there seems to be too little consideration of whether access alone, as emphasized in OER, will also support OOE and promote innovation in teaching and learning (Cronin 2017; Ehlers 2011; OECD 2007). On the other hand, the OOE approach is constantly criticized to deliver education only for privileged learners (Hansen and Reich 2015) or to offer a low-quality educational experience (Czerniewicz et al. 2017; Margaryan et al. 2015). We summarize both sides of the open education continuum in Fig. 1.

Fig. 1
figure 1

Openness in education: OOE focus and OER focus as two ends of a continuum

Full size image

According to Cronin (2017) the use of OER and OOE by educators is contextual, complex, personal and constantly negotiated. Therefore, the true experiences and hesitations of staff and students need to be taken into consideration. Baker (2017) recommends to analyse openness with the current educational model and to let “benefits (and drawbacks) speak for themselves as they are discovered and negotiated”. Since there is a lack of empirical research on educators´ conception of openness for teaching in higher education and the implementation challenges of these two strands of open education, we have conceptualized this study. Rather than focusing on the philosophical and conceptual differences between different types of open education which are often discussed in the literature mentioned above, our aim with this study is to analyse the differences in affordances and the innovation logic of these two types of open education innovation projects in situ.

Thus, the main aim of this study is to explore the identification and prioritization of organizational challenges and opportunities of two subgroups of projects (i.e. OER focused or OOE focused) within various higher education institutions in The Netherlands. Our main research question is: Does the project character (i.e. OER focus vs. OOE focus) of innovation projects lead to perceived differences by actors involved in their implementation? More specifically we will address: Are their differences between challenges and opportunities arising by the implementation of these project based on their specific implementation focus? Our hypothesis for the study was that the differences between the two types of open education projects would also lead to different implementation challenges and opportunities resulting into a need for different implementation strategies.

This research is in line with research traditions like design-based research (Design-Based Research Collective 2003) and implementation science (Fixsen et al. 2005) which have both been established to overcome the inherent theory–practice-gap in (educational) research. Penuel et al. (2011) discuss the combination of both research traditions as “design-based implementation research” which sets itself the goal to focus on persistent problems of practice from multiple stakeholders’ perspectives. In line with this focus we have conducted this research in a number of innovation projects in the Dutch Higher Education system in a funding line by the Dutch Ministry of Education that offered these two implementation strands (i.e. OOE vs. OER).

This study is a follow up study of a previous study by Schophuizen et al. (2018). In this earlier study we have analysed challenges and opportunities for the implementation of open online education based on a sample of initial MOOC projects funded by the Dutch government. The earlier study has helped to identify the obstacles and opportunities that staff and faculty members face who are involved in the implementation of open education. This study was the first study to our knowledge that has been conducted on the implementation level of open education rather than the strategic level. Since the underlying funding program has been adapted, we had access to a larger and more diverse data set that allows us to compare different project types (focus on OOE vs. focus on OER).

The data for the current study stems from 31 funded innovation-projects in the Netherlands aiming to implement OOE or OER initiatives within their respective HEIs. Since these projects are distributed across nearly all universities in the Netherlands, we make the assumption that the results of this study are important and representative for the higher education landscape in the Netherlands and comparable higher education institutions in other western European countries. Next we introduce the method of the study and report and discuss results and implications of results.

Method

Group concept mapping

Group Concept Mapping (GCM) is used to uncover a shared understanding of a group about a certain topic (Waltz 2020; Kane and Trochim 2007; Jackson and Trochim 2002; Trochim 1989). The method is a structured procedure, incorporating quantitative and qualitative variables to create a visual depiction of views about a particular subject that is completely based on the input of participants. GCM has been applied 30 years after its inception by Trochim (1989) to numerous research fields like public health, social work, education or evaluation. The method consists of multiple phases with involvement of different experts The procedure for data collection of GCM consists out of five phases: (1) Preparation (2) Generation of statements (3) Structuring of statements (4) Data analysis, and (5) Data interpretation (see Fig. 2).

Fig. 2
figure 2

[Adapted from: Trochim (1989)]

Flow diagram indicating the process for group concept mapping

Full size image

In phase one the researcher decides on the sample and what the specific focus for the GCM study will be. Deciding on the focus entails the development of a well-defined focus statement for the brainstorm and criteria for the rating of the statements that follow from this brainstorm and takes place in phase two and three. Subsequently the main activities for GCM data collection can start; generation of statements (i.e. brainstorming) and structuring the statements (i.e. sorting and rating).

In phase two, participants produce a set of statements through a brainstorm; these should indicate the complete conceptual domain for the topic of interest. In the brainstorm, participants use the earlier prepared focus statement as a prompt, and they are encouraged to produce many statements freely, without debating about the acceptability of statements that are being produced. During the session, the generated statements are accumulated and made available to see for all participants (e.g. in an (online) tool). After the brainstorm, it is needed to edit the set of statements because analogical statements, statements that contain more than one thought, or statements that are not relevant to the focus prompt should be deleted or rewritten. Also, large amounts of statements put a time burden on the participants in the following phase (i.e. sorting and rating). Therefore, by removing erroneous and double statements, the set can be reduced to the optimum amount of around 100 statements (Jackson and Trochim 2002; Kane and Trochim 2007).

In phase three, the set of statements will be structured by means of sorting and rating. In sorting, participants are instructed to sort the individual statements into piles “in a way that makes sense to them” and to give each pile a title or name. Rating is usually accomplished by using a Likert-type response scale (e.g. a 5-point scale) to indicate how much of the chosen criteria (e.g. importance and influence) is associated with each statement by the participants.

In phase four, the aggregated participant sorting and rating data will be analysed. This analysis takes place in three stages. In stage one, each statement is allocated as an individual point on a two-dimensional point map. Statements that are closely located to each other are sorted together more often; those that are further apart on the map were sorted together less frequently (Trochim 1989). In stage two of the analysis, an hierarchical cluster analysis is done where individual statements on the point map are grouped into clusters. This takes place through an iterative process in which it is decided which amount of clusters are the best interpretation of the data. In general, the goal here is to find the maximum amount of clusters that is still able to meaningfully allocate the statements within those clusters. Subsequently, the average ratings across participants for each statement and cluster is calculated. This data can then be transferred graphically on the maps to create a point rating map and cluster rating maps. This analysis thus results in several products: the point map; the cluster map; the point rating map; and the cluster rating map. These will be used for data interpretation in phase five.

Compared to other methodologies for consensus building, like Delphi or focus groups, GCM has some overlapping features. However, GCM is also able to overcome some of the disadvantages present in these other methodologies. Opposed to the Delphi method, GCM involves only one phase of structuring the data, which is created solely by the participants, not by the researcher. Additionally, GCM also has benefits compared to word-based and code-based techniques concerning reliability (i.e. accuracy, stability, reproducibility) and validity (i.e. sampling validity, construct validity) (Jackson and Trochim 2002; Kane and Trochim 2007). The reason for this is that GCM does not rely on coding schemes that are constructed by the researcher, making the inter-coder discussion unnecessary. Instead, participants do the coding of the text themselves by sorting their self-generated statements (Hynes et al. 2015). More specifically, in GCM the original participant generated statements are used as an observational unit, and subsequently quantitatively aggregates these data through multidimensional scaling (MDS) and hierarchical cluster analysis where consensus is reached voluntary through this multivariate statistical analysis.

The reliability and validity dimension of the GCM method is comparable to mixed method research (Zohrabi 2013). Onwuegbuzie and Johnson (2006) and Collins et al. (2012) provide examples of mixed methods legitimation types with the purpose to ensure that the combination of qualitative and quantitative steps leads to meta-inference quality. Through the systematic process and quality indicators of the GCM method, GCM itself has several checkpoints that signal the meta-inference quality.

Rosas and Kane (2012) have conducted a pooled study analysis to assess the quality and rigor of the GCM methodology. The authors conclude from their study the GCM methodology lead to a strong internal representational validity and strong estimates for sorting and rating estimates. Overall, the authors estimate that validity and reliability of GCM studies are high despite differences in participation rate and task completion. Furthermore, the authors recommend the application of GCM-specific benchmarks for fit and consistency. Several group concept mapping studies have used the procedure outlined by Rosas (2017) to examine the spatial correspondence between the two point maps produced by independent groups on the same content (Rosas 2017).

Participants and procedure

As described above, the procedure for data collection of GCM consists out of five phases: (1) Preparation (2) Generation of statements (3) Structuring of statements (4) Data analysis, and (5) Data interpretation. In the following section we will elaborate on the participants and procedures used in the data collection (i.e. phase 2 and 3) and analysis stage (i.e. phase 4) of our study (see Table 1 for an overview).

Table 1 Step-by-step description of the participants and procedures during data collection and analysis
Full size table

Preparation (phase 1)

In the preparation phase (i.e. phase 1) two main tasks have to be undertaken: (1) the researcher decides who will participate in the study; and (2) the focus for the study, and more specifically the conceptualization needs to be decided on. After these tasks the core parts of GCM can be undertaken: statement generation through brainstorming and structuring the statements through sorting and rating.

For the brainstorming this means there needs to be a formulation of a focus statement. It is key to come up with a clearly defined focus statement, so all participants have a clear idea of what is expected from them. Therefore, the researcher needs to take into account that this statement will not generate double barrelled statements. After defining the focus statement, the criteria for rating the result of the brainstorm should be chosen. In this step it is important to think about what information would be most useful, depending on the purpose of the study.

Brainstorming (phase 2)

For the generation of statements for brainstorming (phase 2) an open call for participation of experts has been shared in several channels (e.g., LinkedIn, Twitter, project websites, affiliated educational websites, personal contact) 59 Dutch OOE/OER experts on (higher) education participated and have provided statements in this phase. It is recommended in the literature that in this phase a diverse set of input from different stakeholders should be allowed to improve external validity (Jackson and Trochim 2002; Kane and Trochim 2007; Trochim 1989). For this phase, the participation of a broad variety of OOE/OER experts ensured that the statements would be a valid depiction of the complete conceptual domain of our topic of interest. We asked individuals involved in the topic of our study, through the aforementioned channels, to contribute with their perspectives on organizational challenges and opportunities for OOE/OER. The individuals we approached comprised generally out of educational researchers and teachers from HEIs in the Netherlands and members of SURF special interest groups—a semi-governmental non-profit organization for promoting and supporting the use of Information and Communication Technologies (ICT) in Dutch HEIs.

For data collection in phase two, we provided the participants with one month to produce statements based on our focus prompt: “My institution has with regard to open online education the following challenge OR chance…”. In total, 149 statements were generated. This set was reduced by the researchers to 106 statements by means of removing duplicates and splitting double premised statements.

Structuring of statements: sorting and rating (phase 3)

For phase three, 33 OOE/OER project leaders participated who were responsible for directing a funded OOE or OER project initiated in 2015, 2016 and 2017 respectively (note: our sample consisted out of 31 innovation projects of which some projects had two project leaders). The participants in this phase were recruited by asking them to volunteer for this study. The involvement of current project leaders ensured a reliable sorting and rating since this group had authentic and recent experience with the innovation projects they were running.

We informed the participants about the purpose, procedure, and the time required to finish the different steps. Additionally, by signing an informed consent they confirmed that they understood the instructions. Subsequently, the project leaders sorted the statements, that were generated in phase two, into groups, naming them with labels that they felt were appropriate. After sorting these statements, the project leaders rated them according to two criteria, namely importance and influence (i.e. Importance; how important is the premise as mentioned in this specific statement to realize this type of education and Influence; how much influence does your institution/organization have on the premise as mentioned in this specific statement). For phase three (i.e. sorting and rating) the data was collected in three separate sessions in face-to-face settings and took an hour and a half in total.

Data analysis (phase 4) and data interpretation (phase 5)

Since the aim is to compare OOE projects with OER projects we divided the dataset (i.e. sorting and rating data) into two groups. The differentiation between the projects was based on the information in their project proposals. If the main idea articulated in the proposal was about developing open learning opportunities, with the involvement of teaching practices and with the inclusion of pedagogical or didactic aspects at scale the projects were distributed in the OOE category. If the main idea articulated in the proposal was about developing content, with an emphasis on open licencing and stressing sharing educational resources with the aims for use and reuse on an individual level the projects were distributed into the OER category.

We analysed both groups separately (i.e. OOE n = 12 vs. OER projects n = 21). The amount of interpretable clusters was defined by using the cluster replay map facility in the GCM online tool, hereby checking for relevance for every suggestion of hierarchical cluster analysis for merging clusters. During all steps, in deciding the amount of clusters, we administered a thorough thematic check to see whether splitting the statements in individual groups was logical based on the content they represent. Lastly, we marked all clusters with meaningful titles based on the titles given by the participants and the content within each cluster, resulting in two cluster maps for OOE and OER projects. The amount of clusters represents the best fit to the sorting data. This means that for a cluster solution with a lower amount of clusters, the sorting data was also less fragmented, whereas for a cluster solution with more clusters there was a more granular sorting. In other words, if participants sort in fewer or more categories, this will also be reflected in the optimal cluster solution.

Using the results from this first analysis, aim of the additional quantitative analysis is to employ a methodologically rigorous approach to determine and subsequently examine the similarities in the conceptual representations generated by the two subgroups in a quantitative manner (OOE vs. OER). Therefore, we hypothesized that the conceptualized patterns, represented by the two multidimensional scaling configurations (i.e., concept maps) are more similar than can be expected by chance alone. We further maintained that although the patterns would be similar, meaningful differences exist. Thus, we sought to identify and evaluate these differences, with emphasis on the implications for interpreting the conceptualized patterns. To meet this aim, a Procrustes comparison approach and permutation strategy will be used to assess the spatial correspondence of concept maps generated by two specific groups outlined by Rosas (2017). Procrustes analysis accounts for the multidimensional nature of MDS configurations and utilizes transformations to find an optimal superimposition between two configurations that maximizes their fit (Gower and Dijksterhuis 2004). Through matrix translation, reflection, rigid rotation and dilation, Procrustes analysis compares the actual ordination results from MDS by matching two configurations of the same objects in a multidimensional space and evaluating the residual variability left in the profile (Schneider and Borlund 2007; Peres-Neto and Jackson 2001). Coupled with a permutation strategy to test the statistical significance of an observed resemblance statistic, the Procrustean comparison approach facilitates the assessment of a multivariate, rather than univariate fit. Furthermore, the approach produces information regarding the optimal superimposition of the multivariate data so that the concordance of the observations for each dimension can be examined separately, aiding in interpretation of the similarities and differences between independently produced concept maps. Interpretation of the analysed data was conducted after phase 4 by the authors.

Results

The project leaders sorted and rated the 106 unique statements. In Fig. 3, a point map of the OOE and OER projects is shown separately. By means of bridging values, calculated by the GCM tool, the location of the points on these maps was construed. These bridging values can take a value between 0 and 1. The higher this value, the further away the statements are grouped from each other, indicating to cover diverging content and vice versa. Thus, statements that are located close to each other on the map, are sorted together more often, resulting in low bridging values, and are therefore considered to be close in meaning.

Fig. 3
figure 3

Point maps of the sorted statements for OOE (top) and of OER projects (bottom)

Full size image

Stress values are an indicator used after MDS to analyse how well the mathematical model, represented by a point configuration in an Euclidian space (point map) matches individuals raw sorting (Cox and Cox 1990). In GCM stress values are used to reflect the goodness of fit of the map to the original dissimilarity matrix that served as input (i.e. sorting data). The lower this stress value is the better the fit. Thus, in order to decide if the point map represents the project leaders' original sorting, we need to consider this stress value. Generally, the stress value for GCM studies should be in the range between 0.205 and 0.365 (Rosas and Kane 2012; Kane and Trochim 2007; Trochim 1989; Trochim 1993). In this study, for both groups this value was within the recognized range. The stress value for the OOE group was 0.2941 after 13 iterations and for the OER group, this was 0.2403 after 15 iterations. Consequently, we could establish that both point maps are an appropriate depiction of the original project leaders' sorting. Furthermore, the configural similarity correlation between sort and distance matrices for OER was 0.78 (p < 0.001) and 0.71 (p < 0.001) for OOE. Both were above the average and at the high end of the range found in studies of similarly constructed concept maps (Rosas and Kane 2012), and differed significantly from each other (z = 8.34, p = 0.000 following Fisher’s r-to-z transformation), indicating a stronger relationship between the sorting and distance matrices of the OER map than the OOE map.

In a first step, separate MDS analysis of the sorting data resulted in 2-dimensional maps with stress values of 0.232 after 11 iterations for OER and 0.294 after 13 iterations for OOE. The stress value of the both configurations were within the range found in previous meta-analytic studies of concept mapping (Kane and Trochim 2007; Rosas and Kane 2012; Trochim 1993). The configural similarity correlation between sort and distance matrices for OER was 0.78 (p < 0.001) and 0.71 (p < 0.001) for OOE. Both were above the average and at the high end of the range found in studies of similarly constructed concept maps (Rosas and Kane 2012), and differed significantly from each other (z = 8.34, p = 0.000 following Fisher’s r-to-z transformation), indicating a stronger relationship between the sorting and distance matrices of the OER map that the OOE map.

By looking at both point maps, we can already recognize several groups of statements. Nevertheless, for spatially separated statements it is more difficult to identify groups. Hence, by means of hierarchical clustering for both OOE and OER point maps we came to a certain optimal cluster solution. For the OOE project leaders this resulted in an eight cluster solution, and for the OER group this resulted in a six cluster solution (see Fig. 4). In Table 2 the cluster labels for both OOE and OER are listed, with an overview of the cluster statement count, average bridging values and range.

Fig. 4
figure 4

Cluster maps of the sorted statements for OOE (top) and of OER (bottom)

Full size image
Table 2 Overview of the OOE and OER clusters
Full size table

As a result of the participant rating, a go-zone graph is constructed (Kane and Trochim 2007). This is a bivariate graph representing the statements in a map where the x-axis represents the criteria influence and the y-axis the criteria importance (see Fig. 5) (Potter et al. 2006). With the help of this go-zone graph we can easily see which statements have a high score on both importance and influence. The statements are positioned in the upper right quadrant, the so called go-zone should have a high impact on the success of OOE or OER, and are also influenceable by the organization. The proportion of clusters that are represented in the go-zone can be found in Table 3. In addition, we have analysed the divided cluster maps for OER and OOE with regard to their statements that either represent challenges or opportunities (see Fig. 6).

Fig. 5
figure 5

Go-zone graphs for OOE projects (top) and OER projects (bottom)

Full size image
Fig. 6
figure 6

Cluster-maps by challenges (red, dotted) and opportunities (green, striped) for OOE (top) and OER (bottom) (Color figure online)

Full size image

Additional results: procrustes comparison

For the first step in the Procrustes comparison, the separate X–Y coordinates from the two point map configurations were entered into the PROTEST software program. The set of coordinates from the OOE point map was held fixed, and the OER configuration was subjected to matrix transformation during the Procrustes analysis, followed by the permutation procedure. The spatial correspondence between the two data configurations was greater than expected due to chance (m2 = 0.35; p < 0.001) with a lower m2 value indicating greater correspondence. Based on King and Jackson’s (1999) categorization of fit scheme for m2 values, 0.35 connotes a higher than average similarity, although caution in applying the categories is warranted since they have not been validated in a sample of concept mapping studies. Nevertheless, the result revealed a highly significant concordance between the multivariate data sets was detected, and the two configurations showed an above average non-random resemblance (Schneider and Boland 2007). However, the result of this first step in the analysis, we cannot say anything yet about the content and distribution across clusters and differentiation between the OER and OOE maps in terms of cluster differences.

Consequently, the differential location of points between the two maps can be observed generally in Fig. 7, illustrating the similarities and differences between the arrangement of the content. Several grouping of points seen on the OOE map were clustered together in a relatively similar pattern in the OER map. For example, those coloured orange in the lower right and those coloured purple in the lower centre of the of the OOE map were in a similar proximal location on the OER map suggesting the two groups organized the content in a similar way and this similarity was consistently plotted in the two-dimensional space. In contrast, points in the top left-hand quadrant of the OER map were distributed with much greater variety, as evidenced by the different colours of points constituting the various clusters in that area. This suggests that those in the OER group organized the content differently in the sorting exercise than those in the OOE group, thereby resulting in a different arrangement of that content in the concept maps (Table 3).

Fig. 7
figure 7

Cluster maps of the sorted statements for OOE (top) and of OER (bottom)

Full size image
Table 3 Proportion of clusters represented by single statements in the go-zone
Full size table

Discussion

Primary outcome measures: cluster maps

Our main research question in our study was: “To what extent do OER projects differ from OOE projects in the identification and prioritization of organizational challenges and opportunities they encounter?”. Our results generated a number of answers to this question and implications that we could understand on the basis of the two cluster maps of both OOE projects and OER projects. For OOE projects the result was an eight cluster solution: (1) Formal quality criteria, (2) Infrastructure and support, (3) Digital skills teachers, (4) Educational innovation, (5) Analysing learning, (6) Educational flexibility, (7) External collaboration and (8) Institutional reputation. For the OER projects the result was a six cluster solution: (1) Teacher role, (2) Teacher support, (3) Effects on students, (4) Sharing knowledge, (5) Educational flexibility and (6) Institutional reputation. We will now compare the two map configurations on several aspects.

Cluster map solutions

For OOE an eight cluster solution had the best fit with the data. For OER a six cluster solution was the best fit with the data. This indicates that for a cluster solution with lower clusters (i.e. OER), the sorting data was also less fragmented, whereas for a cluster solution with more clusters there was a more granular sorting (i.e. OOE). In other words, the OOE project leaders recognized more explicit themes in their sorting opposed to OER project leaders. This suggests that the organizational challenges and opportunities for OOE have a more multifaceted nature, where more sub-themes are of importance and play a role in the implementation and adoption of OOE. The reason for this could be found in the notion that OOE is part of broader social practice with that involves a multidimensional collection of social, technological and educational transformations (Brown 2016). Consequently, implementation of OOE is coupled with these wider considerations of teaching practices compared to OER, which may result in a more detailed set of organizational implementation challenges and opportunities as well. This aligns with findings from a study by O’Connor (2014) where she suggests that MOOCs are not only an external influence stimulating changes to universities, but that they are developed through a complex interrelation of different interests and drivers. Additionally, Tømte (2019) also found that implementing and piloting a MOOC encountered various types of resistance, and that most of this is due to different conceptualizations of what a MOOC is since it is contrasting ongoing teaching activities in a university. In other words, MOOCs and their implementation challenges the existing organizational structure of universities, and requires a bigger change in the core of the universities practices opposed to OER, that has less of an influence on teaching processes.

The results above are exemplified when we consider the differences in differentiation considering the concept of education and pedagogy. In the OOE cluster-map three clusters have been identified which clearly and distinctively describe this theme (Fig. 4, the top map): (4) Educational innovation, (5) Analysing learning and (6) Educational flexibility. If you compare this to the OER map, we can see that there are just two clusters that represents education and pedagogy (Fig. 4, the bottom map): (3) Effects on students and (5) Educational flexibility. Additionally, these two clusters also are very close to each other in terms of proximity on the map. This indicates that the pedagogical aspects, and the aspect of delivering education by means of OER are mentioned, but with little detailed consideration of how this could take form, can be designed for or benefitted from. This goes in accordance with the notion also mentioned by Knox (2013a, b) which states that the OER movement takes self-directed learning for granted, and tends to make assumptions about the capacity for individuals to act purely in an autonomous fashion. This reliance on self-directed learning neglects to address the role of pedagogy within OER.

In the case of OOE projects, the teaching aspect is regarded differently compared to the OER results if we look at the cluster data. The presence of multiple clusters, that are clearly separated on the map could indicate that projects with an OOE focus have a multi-faceted and more detailed view on the nature of education in the case of openness. It shows an explicit distinction between the topics that play a role in the educational and pedagogical concept in which students play a role. Those clusters also are the most coherent in the map (i.e. low range of bridging values). This signals that OOE projects are much more focussed on concrete teaching practices, and opportunities and challenges related to this process opposed to OER, for which teaching is an indirect effect of the content that is created, shared and reused.

On the other hand, in the OER cluster map, there are two separate clusters related to the teacher (i.e. (1) teacher role constraints and (2) teacher support) whereas in the OOE cluster map there is only one cluster related to the teacher (i.e. (3) digital skills teachers). This is in accordance with the OER literature, where it is made explicit that OER is an effort where teachers are the central element in creation, sharing and reuse. It is assumed that in OER projects there is more focus on the teacher because they are central to the creation and lifecycle of OER. Opposed to OOE, where the teacher has a more shared responsibility, and the classical role of a teacher becomes “unbundled” (Buhl et al. 2018).

An additional Procrustes comparison and permutation strategy was used to consider spatial correspondence of the conceptual maps generated by the two individual groups outlined by Rosas (2017). Procrustes analysis takes the multidimensional nature of MDS configurations into consideration and uses transformations to identify an ideal superimposition between two arrangements that maximizes their fit (Gower and Dijksterhuis 2004). This additional quantitative comparison did not lead to additional insights comparing the two maps, and therefore not elaborated on further.

Challenges versus opportunities

Regarding the main challenges, we discovered the clearest results in the upper left clusters for both OOE as for OER projects. For OOE project leaders the clusters with the biggest challenges were: (1) Formal quality criteria, (2) Infrastructure and support, (3) Digital skills teachers since these clusters mainly consisted of statements that represent challenges. For OER project leaders the clusters which were considered the biggest challenge were (1) Teacher role and (2) Teacher support (see Fig. 6). What stands out here is that for both OOE and OER the biggest organizational bottlenecks can be found in the surrounding conditions such as support mechanisms, being either policy related criteria or technical infrastructure related issues confirming results of an earlier study we have conducted (Schophuizen et al. 2018). Moreover, the teacher is seen as an entity that causes challenges for both OOE and OER, stemming from a lack of skills, and unclear role and time constraints in development of OOE and OER.

Apart from challenges, also opportunities were present in the cluster maps. The clusters for OOE project leaders that had a majority of statements representing opportunities were: (4) Educational innovation, (5) Analysing learning, (6) Educational flexibility, (7) External collaboration and (8) Institutional reputation. The clusters for OER project leaders that had a majority of statements representing opportunities were: (3) Effects on students, (4) Sharing knowledge (5) Educational flexibility and (6) Institutional reputation (see Fig. 6). What we see is that both groups recognized the opportunity for their organization to improve the reputation of the institution. All project leaders confirm that through either OOE or OER their institution can generate or signal their brand or value to the outside world. However, it also stands out that OOE sees more opportunities related to learning of students and processes related to improving or innovating education. This contrasts the OER group, which is more leaning towards the sharing of knowledge. What both groups share is that they identified educational flexibility as an important theme, which indicates that for both groups the outcome of either OOE or OER could mean more flexible ways of learning and teaching. However, given the differences between the two groups for other topics, the road towards educational flexibility, requirements in the organization and/or the concrete outcome will look differently, and will require different implementation strategies on an organizational level.

Secondary outcome measures: rating results

With results of the participant rating we identified which matters are important regarding the success of OOE and OER and if HEIs are able to internally influence these matters (i.e. rating criteria: importance and influence). Matters that are rated as important and are also influenceable, should be high on the priority list of HEI's. From Table 2, we can see that the go-zone for OOE is mostly covered with statements that stem from the clusters educational flexibility, institutional reputation, infrastructure & support and external collaboration.

The proportion of statements (i.e. 59.09%) from the cluster educational flexibility that are placed in the go-zone shows that facilitating flexible modes of learning is necessary for the realisation of OOE, and that HEIs should be able to influence this within their organization. Most of these statements describe ways in which OOE can contribute to providing students with time-independent learning, providing flexible learning paths, being able to provide more personalized instruction, providing students with self-paced learning opportunities, learning independent of place and enriching the elective choice in the existing curriculum. Accordingly, the benefit that OOE has to offer in terms of flexible learning solutions is not only an advantage that adds on top of traditional campus education. It is also beneficial for existing traditional curricula. OOE therefore has the potential to positively complement campus education.

Also, the two externally oriented clusters, collaboration outside of the own organisation and establishment of a certain institutional reputation, are regarded as important and influenceable by organizations for OOE success since the statements are well represented in the go-zone (i.e. 45.45% and 55.56%) and mainly contained opportunities. This implies that open online education is very much an effort between institutions taking into account the surrounding context of the organisation, and not solely one’s own campus education or internal structure. This allows also to think about the potential and importance of OOE to innovate outside-in, instead of inside out. When looking at the statements this is mainly enabled due to increased collaborations, exchange of knowledge and expertise and learning between institutions. This benefits knowledge improvement within the own institution by enabling enrichment of their own learning environments through externally gained insights. This, in turn, might also improve the implementation of efficient and appropriate and sustainable infrastructure and support mechanisms within the organisation. This cluster was represented in the go-zone by 46.15% of its statements.

For OER the go zone is mainly dominated by the statements belonging to the clusters teacher role, teacher support and educational flexibility (see Table 2). The cluster educational flexibility is represented in the go-zone by a majority of its statements (i.e. 81.25%) and contains mainly opportunities. By looking at statements it becomes clear that this mainly is about more flexible learning for students by means of giving them more choice options in learning resources and enabling more time and place independent learning. Another topic that stands out is the issue of integrating newly developed materials into existing education and extend the concept of “flipping the classroom”.

On the other hand, the cluster that contains mainly challenges is about the teachers support (i.e. percentage of statements in the go-zone 48.00%). Although advantages for students by OER project leaders are recognized, they also observe a threat in reaping these benefits because the limitation put on the support of teachers dedicated towards developing OER. Support might be required in terms of knowledge on licencing and copyright regulations, technical support and/or platform and more knowledge on the idea behind OER. This might also be dependent on the other cluster represented in the go-zone, with a majority of challenges in the statements: teacher role (i.e. 46.67%). It is clear that the observation is that within organizations there is need to provide or facilitate the teachers to dedicate more time to OER. It could be that this also is related to the lack of supporting mechanisms one of the important matters that hinders the teachers and their available time.

Validity and reliability of the study

Through its process and activities, GCM inherently implements measures that potentially can cope with issues related to validity and reliability. During the idea generation (i.e. brainstorming) experts in OOE and OER representing a variety of perspectives (note: from almost all universities in the Netherlands) produced statements that cover the content domain under investigation. The idea synthesis activity takes care further for the quality of the data before submitting it for the analysis. In idea generation, sorting and rating, the participants work independently to each other to diminish the chance for influencing the outcomes (e.g. the negative effect of ‘groupthink’). It is the advanced statistical techniques such as MDS and HCA that aggregate the individual inputs of the participants and show emerging patterns in data easy to grasp visualisations. A subset of the group of experts with comparable profile to those who took part in idea generation, carried out sorting and rating. There is evidence that about 30 participants is the saturation point above which no significant changes in results take place (Rosas and Kane 2012). Typically in GCM the same number of participants are involved in sorting and rating. However, it should be emphasized that in GCM sorting is the primary activity, rating is the secondary one. A generic indicator for internal representation validity is stress index (i.e. goodness-of-fit). It measures the extent to which the mathematical model (i.e. point map), which is a result of the MDS, reflects the reality (i.e. total similarity matrix) it is supposed to do. The stress index of this project is in the accepted range as found by a meta-analytical study including 69 GCM projects (Rosas and Kane 2012). Given the considerations above we believe that this study would contribute to identifying issues with professional perspectives and institutional strategies for implementing OOE and OER in this particular context of the Netherlands. Perhaps the best way to validate and compare the results outside this context is to replicate the study.

Practical implications and recommendations

The findings indicate that there are differences in conceptual as well as practical representation between the two groups (i.e. OOE and OER) implying that different implementation strategies should be recommended depending on the type of project that is targeted. Although both project types require a dedicated training effort for the teaching personnel involved, there are some distinctions.

The results point to an implementation strategy for OER projects that should focus primarily on the individual teacher and his/her ability, capability and last but not least attitude towards sharing knowledge and information. Attitude has been identified by earlier research as a barrier for the adoption of OER in HEIs (Rolfe 2012; Kreijns et al. 2013). Especially this attitudinal dimension requires much deeper approaches to training and professional development compared to the often technical focus which training programs for faculty members in OER have. For this purpose, a longer-term strategy would be required to implement OER into HEIs. We envision professional development programs which focus on the one hand on teachers skills to produce, use and reuse OER but at the same time deal with knowledge- and information sharing inside and outside their institution. Besides concerted training efforts, internal communication and support would complement a systematic strategy. It is recommended to take the specific organizational context under serious consideration, assess what the needs of teachers are, before any programs, trainings or support is being designed.

For OOE projects, the implementation strategy should be more directed towards an institutional effort involving quality assurance mechanisms, an agreement on educational innovation and last but not least infrastructure and support. Opposed to OER, the implementation strategy for OOE is much broader in an organizational sense and “touches” much more processes, structures and stakeholders within the organization. While OER implementation strategies should focus on the individual, OOE implementation strategies ensure that the surrounding context for the development of open educational formats is developed inside the institution. From a managerial perspective, such an effort requires a more holistic change initiative in which all the parts that are involved within the HEI are incorporated into the solution or improvement plan.

Limitations and direction for future research

This study has some limitations. While the overall amount of participants for this study was acceptable, the guidelines on the ideal number of participants for the sorting and rating of statements (i.e. 20–25 individuals according to Trochim 1989) could not be reached for the OOE group contributing to a potential lower variability of results. Subsequently, future research should be directed at uncovering the variables, aspects and mechanisms that have impact on the challenges and opportunities for OOE and OER in more detail. In this sense the results of this study are valid only on an aggregated level and cannot be used to identify specific challenges and opportunities for HEIs separately.

Moreover, since the participants in our study were from Dutch HEIs, who were granted funding for their projects, the findings of our study should be applied to comparable settings. Projects with access to funding have a benefit over non-funded initiatives and projects. The function of these subsidised projects nonetheless is to be able to cultivate best practices, and to build more understanding and awareness within their context that could help with overcoming the challenges for initiatives that are being initiated without funding support.

Conclusion

In this study, we studied to what extent OER projects differed from OOE projects regarding the identification and prioritization of organizational challenges and opportunities within the context of Dutch HEIs. An inventory of the categorization and priorities was made by using group concept mapping. The main themes regarding the challenges and opportunities of OOE projects were: (1) Formal quality criteria, (2) Infrastructure and support, (3) Digital skills teachers, (4) Educational innovation, (5) Analysing learning, (6) Educational flexibility, (7) External collaboration and (8) Institutional reputation. For the OER projects these were: (1) Teacher role, (2) Teacher support, (3) Effects on students, (4) Sharing knowledge, (5) Educational flexibility and (6) Institutional reputation.

Our findings show that there are differences in conceptual and practical representation between the two groups implying that different implementation strategies should be recommended depending on the type of project that is targeted. By a rating component in our methodology we could also gain insight on the priorities of these challenges and opportunities that were experienced by the two groups (i.e. OOE and OER projects). The findings imply that there is currently still a long road ahead within HEI's to fully benefit from OOE and OER projects and to capitalize on the opportunities that both have to offer. This also needs an understanding of the differences between OOE and OER projects in terms of prioritization. Additional empirical research to uncover in what way HEIs could organize this and what mechanisms support or hinder this is necessary.

Although openness in education is often used as an umbrella term, it should be considered what exactly is meant by “openness in education” as it seems, according to our findings, that it can contain various meanings and underlying assumptions geared towards different goals and results. More specifically, this study was able to identify empirically some of the differences in organizational requirements in order to implement either OER or OOE projects. In other words, “openness” on its own does not seem to reflect a universal educational value and needs to be placed in context to be understood. With this study we aim to contribute to placing openness in education in context, and shed light on the differences, not by means of theory, but with empirical evidence.