Developing, Validating, and Implementing a Tool for Measuring the Readiness of Medical Teachers for Online Teaching Post-COVID-19: A Multicenter Study

Background

No matter how well-prepared, medical schools all over the world were recently thrown off balance when they were forced to shift to online teaching. With no end to the COVD-19 crisis in sight, schools have realized that they are unlikely to return to normal anytime soon, and that online teaching has moved from adjunct learning method for both teachers and learners to a main modality. Teachers around the world have had to move their courses online in the face of several challenges, namely the need to (1) promptly shift to online teaching, with little to no preparation; (2) implement and maintain online teaching under the difficult pandemic conditions; and (3) follow extended online teaching with little to no information regarding the expected duration of the shift.¹

Online teaching is an eminent advancement in education in the twenty-first century, where online instructors play an essential role, necessitating being well-trained and adaptable to technology.² In normal situations, it is recommended that an appropriate amount of time is allocated to academics to first acquire the new pedagogical and technical skills before actual operation within an online classroom. Several considerations are important in online teaching including design of online courses, the skills of online instruction and student engagement, and the appropriateness of online delivery to the respective discipline.³ Other considerations that might influence teachers’ readiness are years of teaching experience, self-efficacy with regards to communication and learning transfer, as well as the teacher’s self-directed learning skills.⁴

Moreover, it was reported by Eslaminejad et al that faculty should continuously be offered training to upgrade their information technology knowledge and skills over time. In addition, their study results indicated that pedagogical innovations are required to develop and implement an effective e-learning program.⁵

Great debate between researchers exists concerning the optimal toolkit of competencies required by instructors for online teaching. Some educationists argue that these competencies are not significantly different from those necessary for face-to-face teaching;⁶ and it is assumed that past teaching experience is sufficient to teach online.⁷ However, others disagree and support their argument by the fact that the toolkit needed by online teachers differs in 3 main issues; focusing on teaching time and space, learning management systems techniques, and the skill to engage students.^8,9 More recently, Martin et al⁹ described four areas for online teaching instructors’ competencies including course design, course communication, time management, and technical skills.

Some investigators studied the perception of medical educators towards online teaching. Downing and Dyment¹⁰ examined educators’ perceptions of preparing teachers for a purely online environment and found that online teaching was felt to be time-consuming. Northcote et al¹¹ revealed that faculty felt low self-efficacy in selecting technological resources and high self-efficacy in online course alignment (effectively aligning objectives to learning activities and assessments). For successful conduct of online learning, teachers need to put themselves in the position of self-learners when it comes to the required competencies. Thus, understanding teachers’ readiness as learners to engage in online learning not only enables instructional designers to deliver better online courses, but also enables educational institutions to better help teachers improve their online learning experiences.⁴

In the context of the COVID-19 crisis, multiple factors can impact faculty readiness and competencies regarding online teaching. It was claimed that the urgent need for readiness produced positive attitudes among teachers, including their willingness to revise their teaching for online delivery as well as to share control in their classrooms with students whose technological expertise exceeded their own.¹ There is a need to move beyond an instrumental approach to online teaching and learning and define the roles and responsibilities of educators in online education and their impact on the process of learning to teach. This includes the ethical and political dimensions and the consideration of issues of power and control over teaching and learning.¹²

We claim that this study makes an important contribution in unraveling the complexities of online teaching. The educational transformation into the online mode is dependent on the level of readiness in many areas that are explored in this study. Thus, the aim of this study is to develop, validate, and implement a tool for measuring the readiness of medical teachers for online teaching in three different medical schools in three countries in the Middle East and North Africa (MENA) region. This would be helpful for the three involved medical schools to tailor their faculty development programs according to the responses of its faculty members. Additionally, other medical schools worldwide can use our validated tool to check online teaching readiness of their educators.

Subjects and Methods

Type of Study

A comparative, cross-sectional, survey-based study.

Study Setting

This is a multicenter study that included three medical schools, which are the Faculty of Medicine, Suez Canal University, Egypt (FOM-SCU); the College of Medicine, Imam Abdulrahman Bin Faisal University, Kingdom of Saudi Arabia (COM-IAU); and the College of Medicine and Medical Sciences, Arabian Gulf University, Kingdom of Bahrain (CMMS-AGU). Before the COVID-19 pandemic, the three schools were depending only minimally on online teaching/learning, but after the pandemic stroke there was a sudden complete shift to the online mode. The curricula of the three schools are similar where all of them are innovative curricula that employ problem-based learning with both vertical and horizontal integration of basic and clinical sciences. Faculty characteristics and faculty workload are similar across the three schools. In addition, Moodle learning management system is used in two of the schools, while the third school uses Blackboard^®.

Sampling

Study Population

The study population included male and female teaching faculty members of different academic ranks (professors, associate professors, assistant professors, and lecturers) at the three study centers. Teaching assistants and lab technicians were not included in the study as they do not have online classes.

Sample Size

The study used a convenient sample of teaching faculty members including all academic ranks (professors, associate professors, assistant professors, and lecturers). The total number who responded to the questionnaire within the timeframe given (one month) was 217 from the three medical schools (around 24% of the teaching faculty members at the three schools collectively).

Instrument

The survey was developed by the researchers following the steps of 1) extensive review of the relevant literature and similar studies that included items that address the readiness of teaching faculty members for online teaching, 2) item development and revision by all authors, 3) revision by five medical education experts from the three schools, and 4) pilot testing on a small number of respondents. The survey was written in English language and included 30 items representing basic technology skills, communication skills, the skills of using learning management systems, online teaching skills, and course planning skills. It started as a 5-point Likert scale (Strongly Agree, Agree, Uncertain, Strongly Disagree, and Disagree); however, for collecting clear responses from the study participants, the 5-point scale was reduced to a 3-point scale, where “Strongly Agree” and “Agree” were considered as “Agree”, while “Strongly Disagree” and “Disagree” were considered as “Disagree”. This 3-point scale was used to collect data from the participants.

To determine the suitability of the survey, validity and reliability studies were conducted. Two types of validity were established for the survey. The first was content validity thorough revision by a group of medical education experts from the structural aspect and the different dimensions and the ability of questions to explore the readiness of faculty members for online teaching. The second was construct validity through Exploratory Factor Analysis (EFA). The survey was tested for reliability (internal consistency) through calculating Cronbach’s alpha value.

Data Collection

The questionnaire was converted into an electronic format using SurveyMonkey^® and distributed to the target population in each medical school through different communication platforms (like teaching faculty members forums, WhatsApp^® groups, official e-mails…). At the beginning of the online survey, the respondents were briefed about the aim of the study and were given the liberty not to respond to the survey, without any consequences. Each respondent was given the chance to respond only once to the questionnaire. The questionnaire was open for data collection for a period of one month, after which no responses were allowed.

Data Analysis

The statistical analysis has been performed using the Statistical Package for Social Science (SPSS) for Windows, version 25. Data collected through the questionnaires were presented in the form of frequencies.

For comparing the mean scores of the three schools, means and standard deviations were calculated and analysis of variance (ANOVA) was used. A p-value ˂ 0.05 was considered statistically significant.

Internal consistency was analyzed using Cronbach’s alpha. Missing data were treated by replacement with mean of missing variables.

Testing the psychometric properties of the questionnaire was performed through EFA. To identify the different factors, EFA was performed using principal component analysis with Varimax rotation. The number of factors extracted and used was based on the Kaiser criterion, which considers factors with an eigenvalue greater than one as common factors,¹³ the Scree test criterion (the Cattell criterion) to identify the inflexion point indicated by the Scree plot,¹⁴ and the cumulative percentage of variance extracted (in humanities research, the explained variance is usually only 50–60%).¹⁵

After applying the psychometric criteria, the retained factor solutions were then analyzed according to the following interpretability criteria:¹⁶

• A given factor contains at least three variables with significant loadings, with a loading of 0.30 suggested as the cutoff.
• Variables that load on the same factor have the same conceptual meaning.
• A variable that loads on a different factor measure a different construct.
• The rotated factor pattern shows a “simple structure”, meaning that:
  • Most variables load relatively high on only one factor and low on the other factors.
  • Most factors have relatively high factor loadings on some variables and low loadings on the remaining variables.

Ethical Approval

The study was approved by the Research and Ethics Committee (REC) of the College of Medicine and Medical Sciences, Arabian Gulf University, Kingdom of Bahrain (No. E004-PI-9/20). All survey participants provided informed consent.

Results

The results are divided into two parts:

Part I: Validity and Reliability of the Newly Developed Survey

Results of the Validity Study

According to the review done by five medical education experts from the three schools, modifications to some items were made and the survey was edited and made ready for administration to the study participants. Examples of the made modifications were adding examples of web browsers, learning management systems, and synchronous online teaching platforms.

The collected responses were 217, which is adequate for factor analysis. Analysis of data adequacy indicated an adequate amount of data for factor analysis (Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) was 0.91 and Bartlett’s Test of Sphericity χ² (435, N=217)=3379.63, p=0.00). Therefore, this output indicated the appropriateness of the data for factor analysis.

Results of factor extraction revealed that the 30 items of the survey could be grouped under five factors with an eigenvalue >1.00. The five factors that emerged from factor analysis accounted for 58.1% of the total variance.

Results of factor rotation showed that none of the 30 items of the survey were removed from the analysis. This was based on finding that a) all the factors had three or more items, b) no items had cross-loading between factors, and c) all items had a loading of >0.30 on the relevant factor.

After conducting the previous validity studies, the survey was composed of the original 30 items distributed to 5 factors. The factors were named according to the heaviness of loading of the statements (items) on each factor and based on the idea behind the statement (Table 1) as follows:

• Factor 1 explained 35.63% of the variance in responses, with an eigenvalue of 10.69. Twelve statements loaded on this factor, with values between 0.45 and 0.72. This factor has been renamed to “Online Teaching and Course Design Skills”. This factor addresses the skills of designing courses and educational materials for online teaching and learning, excelling in online teaching, and seeking development in such skills.
• Factor 2 explained 8.52% of the variance in responses, with an eigenvalue of 2.56. Six statements loaded on this factor, with values between 0.43 and 0.68. This factor has been renamed to “Digital Communication”. This factor addresses the confidence in communicating verbally and in writing and giving feedback to learners.
• Factor 3 explained 5.68% of the variance in responses, with an eigenvalue of 1.70. Five statements loaded on this factor, with values between 0.50 and 0.80. This factor has been renamed to “Basic Computer Skills”. This factor addresses the skills of file management and document creation using the applications of Microsoft Office, sending and receiving emails, surfing the internet for educational materials, and being familiar with a learning management system.
• Factor 4 explained 4.19% of variance in responses, with an eigenvalue of 1.26. Three statements loaded on this factor, with values between 0.59 and 0.76. This factor has been renamed to “Advanced Computer Skills”. This factor addresses the skills of file encryption and recording audio and video clips.
• Factor 5 explained 4.08% of variance in responses, with an eigenvalue of 1.23. Four statements loaded on this factor, with values between 0.61 and 0.69. This factor has been renamed to “Using Learning Management Systems”. This factor addresses the comfort and confidence in using learning management systems in course development and management.

Table 1 Factor Loadings of the Items Under the Five Factors of the Survey Form (Using Principal Components Analysis)

Results of the Reliability Study

Test of reliability (internal consistency) revealed high reliability of this tool (Alpha = 0.94).

Part II: Survey Results

Two hundred and seventeen faculty members from the three schools responded to the survey. As shown in Table 2, the profile of the respondents revealed that the majority of them are in the age group of 46–55 years at the FOM-SCU and COM-IAU, while at the CMMS-AGU the majority are in the age group of 56–65 years. Most of the responses were from females at FOM-SCU and COM-IAU (63.3% and 69.8% respectively), while this was reversed at CMMS-AGU where most of the respondents were males (61.1%). The academic rank “full professor” was the most prevalent among FOM-SCU respondents (45.8%), while “assistant professor” was the most prevalent among CMMS-AGU and COM-IAU respondents (33.3% and 46.6%, respectively).

Table 2 Profile of the Respondents of the Three Participating Schools

The Results of Teacher Responses Under Each Factor are Presented Hereafter

Regarding Online Teaching and Course Design Skills (Table 3), the great majority of instructors in our sample expect online teaching to take more time and they are ready for that. They feel comfortable conducting interactive learning activities and creating online teaching materials. They agreed that they can create schedules for themselves and stick to them. Some differences were noticed between the three schools with respect to faculty perceptions of their readiness for online teaching.

Table 3 Instructors’ Perception of Their Readiness in Regard to “Online Teaching and Course Design Skills”

On the other hand, most instructors incorporate online learning activities that are connected to real-world applications during teaching and are always keen to attend as learners in online faculty development activities. They are oriented with online course planning, comfortable writing measurable learning outcomes, and understand copyright law and Fair Use guidelines when using copyrighted materials in education. Some differences were noticed between the three schools, which were not consistently toward any of them.

Two-thirds of instructors agreed on three items (enjoying online lecturing for most of the class period, knowing how to check for plagiarism in students’ assignments, and feeling comfortable designing online interactive learning activities). Some differences were noticed between the three schools, which were not consistent toward any of them.

For almost all items, the least percentages were for those who disagree, while no more than one-fifth of participants were uncertain about their perception of any of the items.

Regarding Digital Communication (Table 4), the great majority of instructors in the three schools agreed that they feel comfortable communicating through writing and speaking and they are willing to provide timely and constructive feedback to their students. Some differences were noticed between the three schools, which were not to the side of any of them.

Table 4 Instructors’ Perception of Their Readiness in Regard to Digital Communication

For the other three items (feeling comfortable using social media tools to communicate with students and colleagues, being ready to timely respond to communication requests from students and colleagues and being available to students on a regular basis for questions and assistance), most of the instructors agreed.

Regarding Basic Communication Skills (Table 5), almost all the instructors agreed that they can use Microsoft Office tools such as Word and PowerPoint to create documents and presentations. They can perform file management on their computers, such as copying, moving, renaming, and deleting files or folders. They can also send and receive emails, including opening and sending email attachments. They can use internet browsers to locate resources for teaching; and they are familiar with at least one synchronous online teaching platform. Minimal statistical differences were noticed in the perception scores between the three schools’ respondents.

Table 5 Instructors’ Perception of Their Readiness in Regard to Basic Computer Skills

Regarding “Advanced computer Skills” (Table 6), the great majority of the instructors agreed that they can encrypt files on their computers, record audio/video, and add audio/video files to presentations. Some differences were noticed between the three schools, which were not consistently in favor of any of them.

Table 6 Instructors’ Perception of Their Readiness in Regard to Advanced Computer Skills

Regarding Using Learning Management Systems (Table 7), the majority of the instructors agreed that they are comfortable using learning management system tools to develop online courses, using tools in the learning management system to facilitate student learning, and using the learning management system or other online assessment tools to evaluate student performance. Some differences were noticed between the three schools, which were not consistently in favor of any of them.

Table 7 Instructors’ Perception of Their Readiness in Regard to Using Learning Management Systems

Table 8 shows the factors’ mean scores for the three schools. The mean scores for Basic Computer Skills were the highest for the three schools, while those for Online Teaching and Course Design Skills and Using Learning Management Systems were the lowest for the three schools. Analysis of variance of the responses revealed that there are statistically significant differences regarding two of the factors, which are Digital Communication (F=5.13; p=0.007) and Basic Computer Skills (F=4.47; p=0.012).

Table 8 Analysis of Variance (ANOVA) of the Mean Scores of Responses in the Three Studied Medical Schools

Discussion

The value of measuring the readiness for online teaching emanates from the fact that online instructor readiness plays a key role in the success of e-learning,^17–19 partly because perceived self-efficacy has a high impact on perceived ease of use.¹⁸

The tool designed in this study is a 30-item Likert scale (Online teaching readiness questionnaire – OTRQ) that measures teachers’ readiness for online teaching. To the best of our knowledge, this is the first comprehensive tool to measure readiness of teachers for online instruction in the Middle East. The reliability of this tool was tested on >200 faculty and proved to be excellent, with a Cronbach's alpha value of 0.94 (acceptable values range from 0.70 to 0.95).²⁰ Previous studies have shown reliabilities ranging from 0.70^18,19 to 0.86.²¹

Validity study through EFA yielded five components (factors) that give a holistic readiness estimate combining the three phases of pre-course, during the course, and after the course, and stresses on the technical and pedagogical factors involved in such state of readiness. We used a 3-point Likert scale, which is nearly similar to other studies that used 3-point,²² 4-point,^23–25 5-point,^2,26–30 and 7-point^18,31 Likert scales. The number of items was 30 in OTRQ, while in other studies it varied from 18 to 49.^{2,25–27,32–39} Our validity study yielded 5 factors as previously described, while the number of factors in other studies varied from 3³³ to 14³² factors. The number of participants in our study was 217, which was sufficient to validate the instrument. Other validation studies used varied numbers of participants, ranging from 20⁴⁰ to 369.³⁴

The validation process used in this study included content validation through experts’ opinions and construct validation through EFA. All previous studies had to discard some items based on the criterion of low loading or cross loading with different factors, but in this study, no items were discarded as every item fulfilled the criterion of having a loading of >0.3 on a particular factor. Most studies depended only on content validation through expert revision, correlation, and literature review. Only three studies of online teaching readiness measurement tools employed factor analysis.^27,37,41

Age was found to influence faculty perceptions towards e-learning with the younger age being associated with more positive e-learning readiness and less technophobia.^1,2 The respondents belonged to three different countries in the MENA region and to long standing nationally accredited higher education institutions. Almost all (91%) had a rank of assistant professor or higher. As faculty rank increases, the perceived ease of use and readiness for online teaching score decrease. However, this relationship was found to be not statistically significant. This agrees with findings from a study by Martin et al,⁹ who found that full and associate professors were inferior to assistant professors and lecturers in both attitude and ability of online course design, online course communication, and technical competence.

The scores obtained from the three schools showed a high level of online teaching readiness among instructors. The readiness was more on technical and attitude aspects than on pedagogical skills. It showed an appreciative level of overall teacher readiness for online instruction of 85.4%, which is in agreement with previous studies.^2,26,42 Other studies have shown much lower levels of teacher readiness for online instruction of 20%,⁴³ 28%,² 34%,⁴⁴ and 54%.⁴⁵

In our study, the lowest score for all schools regarding Online Teaching and Course Design was for the statement “During teaching, I incorporate online learning activities that are connected to real-world applications. (i.e., using real clinical cases, reflecting on applying knowledge in life uses… etc.)”. This probably means that instructors do not have experience in creating online instructional materials taken from real-life examples or they do not have the sufficient time or facilities to do that. Training and assistance on e-learning is one solution for this problem as suggested in previous studies.⁴⁶ Added to this, the type of learning management system platform influences instructors’ perception of readiness for online teaching.¹⁸ This is on par with previous studies that emphasized that the main obstacle to online teaching was the lack of pedagogical or online course design skills.^2,18,33,34

The gap between the positive digital and computer skills perception and the lack of use of electronic learning management systems found in this study is similar to previous studies which showed that the lack of time and the high workload in addition to lack of recognition of e-learning material preparation were the reasons behind this gap.^25,26,45

Analyzing previous studies on instructor readiness for online teaching showed that there is a lack of standardization among them, including inadequate methodology on exploring the readiness for online teaching scores, and lack of proper statistical validation methodology. Readiness tools for online instruction have produced results and identified problem areas, but have not provided solutions to address deficiencies. Furthermore, these tools did not actually measure the knowledge, skills, and attitudes required by users in online teaching but instead broadly explored perceptions of instructors about their own technical skills and behavioral categories to predict readiness for online teaching.^19,46 In other words, there is a gap between measurement of faculty perceptions and the actual use of e-learning.⁴⁶ Moreover, the tools that assess readiness for online teaching consist of items and domains that are appropriate for use in developed countries and international institutions, but not necessarily appropriate for developing countries and local institutions.¹⁹ The current study helped address this gap by exploring the degree of readiness of medical teachers for online teaching through their knowledge, skills, and attitudes, and by evaluating how such behaviors impacted their level of use of the learning management system during delivery of an online course.

Among the three schools, the highest mean scores for Digital Communication and Basic Computer Skills can be explained by the fact that such skills are used frequently by faculty members, as they communicate all the time through different means of digital messaging and electronic emailing and all of them use computers to create and manage files like presentations and handouts. This in addition to the regular training provided by the three schools’ IT units in basic computer skills. The work of Eom et al⁴⁷ on the role of information technology in the success of e-learning supports this view. The lowest mean scores for Online Teaching and Course Design Skills and Using Learning Management Systems can be explained by the fact that such skills are relatively new to them and were uncommon among faculty members in the three schools that depended mainly on face-to-face teaching before the COVID-19 pandemic. This can highlight these two areas as priorities for future faculty development programs. Moreover, the statistically significant difference between the three schools with regard to Digital Communication and Basic Computer Skills indicates the individual differences between people in using computers and digital communication applications and platforms.

The study has a couple of possible limitations. First, using two different online learning management systems in the three schools maybe a confounding factor as there expected to be differences in the user-friendliness of the two systems. Second, nature and efficiency of technological support and faculty development provided to teaching faculty in the three schools is expected to be different.

Conclusion

Measuring readiness for online teaching is important, especially in a situation of physical distancing like the one we are facing because of COVID-19.

A highly reliable and valid tool was developed in this study tool, which can be implemented by other medical schools. Analysis of the results indicated an overall acceptable readiness of teachers for online learning in the three involved medical schools. However, we recommend faculty development in Online Teaching and Course Design Skills as well as in Using Learning Management Systems as important areas that affect the readiness of medical teachers for online teaching.