Accessibility of university websites worldwide: a systematic literature review

Abstract

The identity and institutional image of universities are presented to the world through their websites. On their websites, universities publish their academic offerings, their mission, their vision, their academic objectives, their achievements, their regulations, their news and all their university work. Hence, the importance of university websites is accessible. The accessibility of university websites has been evaluated several times in the past, but there is no work that has summarized all the evaluations performed to provide a general overview of the situation. Therefore, in this research we have performed a systematic literature review (SLR) to consolidate, analyze, synthesize and interpret the accessibility results of university websites published in 42 papers that have been selected for this study. The methodology used in this SLR was that proposed in Kitchenham’s guidelines, which includes three stages: planning the review, conducting the review and reporting the review. The results present the analysis and synthesis of the evaluations of 9,140 universities in 67 countries. Of these, 38,416 web pages, 91,421 YouTube videos and 28,395 PDF documents were evaluated. Manual methods, methods with automatic tools and the combination of both methods were used for the evaluation. Most websites were evaluated using the ISO/IEC 40500:2012 and Section 508 standards. The accessibility guidelines most commonly violated in the evaluations were: adaptable, compatible, distinguishable, input assistance, keyboard accessible, navigable, predictable, readable and text alternatives. In conclusion, the university websites, YouTube videos and PDF documents analyzed in the 42 papers present important accessibility problems. The main contribution of this SLR is the consolidation of the results of the 42 studies selected to determine the findings and trends in the accessibility of university websites around the world.

1 Introduction

The SARS-COV-2 pandemic has changed and revolutionized the world, making human beings interdependent on technological equipment whether for work, education, health, acquisition of essential supplies, among others. Also, large-scale national efforts to implement technology in all the processes that humans perform are emerging and evolving rapidly. In addition, electronic equipment, the Internet and the Web play an important role in this process of change. People with disabilities are no strangers to this transformation, but in many cases they must face new barriers rather than experience benefits.

In the Convention on the Rights of Persons with Disabilities (CRPD) of the United Nations [1], access to information and communication, including the Web, is defined as a basic human right. In the Article 21—Freedom of expression and opinion, and access to information, it is stated that governments should urge “private entities that provide services to the general public, including through the Internet, to provide information and services in accessible and usable formats for persons with disabilities” [1].

Tim Berners-Lee, inventor of the World Wide Web and Director of the World Wide Web Consortium (W3C) [2], states that “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect” [3]. The W3C’s Web Accessibility Initiative (WAI) stipulates that “it is essential that the Web be accessible to provide equitable and equal access to people with diverse capabilities and not exclude people from using its products and services” [4]. The W3C has created the Web Content Accessibility Guidelines (WCAG) to make websites, electronic documents, PDFs, videos and other resources accessible to people with and without disabilities. It should be noted that WCAG 2.0 was declared as the international standard ISO/IEC 40500:2012 [5, 6]. Therefore, web accessibility applied to university websites will enable people with disabilities to use the Web, including people with “blindness or low vision, deafness or hearing loss, movement limitations, speech disabilities, photosensitivity and combinations thereof, and some adaptations for learning disabilities and cognitive limitations” [7].

In the 2011 World Disability Report, the World Health Organization (WHO) considers that “more than a billion people are estimated to live with some form of disability, or about 15 % of the world’s population (based on 2010 global population estimates). This is higher than previous WHO estimates, which date from the 1970s and suggested around 10 %” [8, pp. 7]. Worldwide, the increase of people with disabilities is notorious according to the statistics of WHO, which allows to affirm that millions of people with disabilities are studying at universities right now. The admission of people with and without disabilities in regular classrooms generates obligations that university institutions must comply with, for example, the accessibility of their web platforms and their educational resources.

Article 24 Education of the CRPD [1] requires the inclusion and participation of all persons in the educational environment. Education is an acquired right that every human being has. Schools, colleges, institutes and universities have been forced to close their facilities, adapting them to online education due to the SARS-COV-2 pandemic [9]. This change has brought with it many barriers that can influence the teaching–learning process of students. These barriers may be reflected in the mastery of technology by teachers, parents, students and the accessibility of the platforms used for the teaching and learning process. The evaluation of the accessibility of university websites has been carried out in numerous research studies to determine their compliance with WCAG, accessibility laws and standards. However, there is no study that analyzes all the work done to give an overview of their situation.

This review examines a dataset resulting from the evaluation results of the accessibility of university websites. The objective of this systematic literature review (SLR) is to synthesize the accessibility results of university websites: to make an analysis, synthesis and interpretation of the results published in 42 selected papers, to determine the web accessibility standards, evaluation methods used and the results obtained. The SLR methodology has been used to guide the research process [10].

Before carrying out this SLR, we made sure that no other similar research study exists [11]. To corroborate this, an exhaustive search for SLRs was carried out to determine the need for it. An SLR establishes a solid base to improve the knowledge, promotes the development of the theory, closes the areas that have been studied in excess and reveals the areas that need to be investigated [12]. Therefore, an SLR is important in determining the application of WCAG and accessibility barriers on university websites worldwide.

This SLR includes the following sections. Section 2 provides the background needed to understand the WCAG, its principles, guidelines, checkpoints or success criteria and their conformance levels. In Sect. 3, the methodology that will help to achieve the goals of the SLR is detailed. In Sect. 4, the results of the research questions and sub-questions are analyzed, synthesized and interpreted. In Sect. 5, the discussion highlights the most important findings of this SLR in an orderly and logical manner and identifies trends and gaps. In Sect. 6, the limitations of this study are presented. Finally, Section 7 presents the conclusions and future work.

2 Background

This section is necessary to interpret the results obtained from the SLR. It describes the recommendations of WCAG 1.0, 2.0 and 2.1, with their principles or priorities, guidelines, success criteria or checkpoints and conformance levels. The recommendations of the WCAG guide developers in building more accessible websites for people with and without disabilities [13]. In addition, the WCAG also explains how to make accessible the content that will be published on the websites (PDF documents, videos and so on). These guidelines are also used to assess the level of accessibility of websites.

2.1 Web content accessibility guidelines (WCAG) 1.0

WCAG 1.0 [14] was published in May 1999 by the W3C, with the aim of providing solutions to web accessibility problems. WCAG 1.0 includes 3 priorities, 14 guidelines, 65 checkpoints and 3 conformance levels, as shown in Fig. 1. Each checkpoint has a priority level according to its impact on accessibility. In addition, each checkpoint has a conformity level that allows to define the conformity level of a website.

Fig. 1

Priorities, checkpoints and conformance levels WCAG 1.0

2.2 Web content accessibility guidelines (WCAG) 2.0 and 2.1

The WCAG 2.0 [15], published in December 2008, includes 61 success criteria that are organized under 12 guidelines and 4 principles: Perceptible (P), Operable (O), Understandable (U) and Robust (R).

The WCAG 2.1 [7], published in June 2018, includes the WCAG 2.0 principles, guidelines, success criteria and conformance levels, plus 17 new success criteria and a new guideline. For a website to fully comply with the recommendations of WCAG 2.1, it must meet all the success criteria of WCAG 2.1 and conformance levels. Therefore, when a website is compliant with WCAG 2.1, the website is also compliant with WCAG 2.0. Figure 2 presents the principles, guidelines and conformance levels of WCAG 2.0 and 2.1. In addition, the 17 new WCAG 2.1 criteria have been highlighted in blue. The W3C is working on WCAG 2.2, and a draft was published in August 2020 [16].

Fig. 2

Principles, guidelines and conformance levels of WCAG 2.0 and 2.1 [17]

2.3 Conformance levels

WCAG 1.0, 2.0 and 2.1 have three conformance levels [7]:

• A: “(the minimum level of conformance), the web page satisfies all the level A success criteria”;

• AA: “the web page satisfies all the level A and level AA success criteria”;

• AAA: “the web page satisfies all the level A, level AA and level AAA success criteria.”

2.4 Website accessibility conformance evaluation Methodology (WCAG-EM)

The procedure for evaluating websites using WCAG 2.0 developed by the W3C/WAI is the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) [18]. This methodology guides evaluators to use good practices to avoid common mistakes and achieve more reliable results. The steps of this methodology can be seen in Fig. 3.

Fig. 3

Evaluation Procedure: WCAG-EM 1.0 [18]

2.5 Evaluation methods

Evaluation methods define the procedures, evaluation tools, end users and experts who assist in the evaluation of university websites. In an SLR carried out in 2019 [19], on web accessibility evaluation methods, it was determined that the main methods used are: “1) automatic tools, 2) evaluation by experts and 3) user tests are the most widely used techniques according to the literature.”

A list of tools for reviewing the accessibility of web content, either manually or automatically, is available on a website maintained by the W3C [20]. These tools are online services or software programs (AChecker, TAW, WAVE and others) that assist in checking whether web content is WCAG compliant. In addition, on this website we can find detailed information about each evaluation tool and its access link. Also, this website provides a set of filters to help the user find the evaluation tools that best suit their needs. Developers can use free or paid tools for web accessibility evaluation, or a combination of both. However, WebAIM [21] states that “no automated evaluation tool can tell you if your site is accessible, or even compliant. Human testing is always necessary because accessibility is about the human experience.”

On the other hand, evaluations with real users [22] are informal or formal experiments that are carried out with people who navigate a website in a normal way and their behavior is observed by the evaluators. After these experiments, the evaluators determine the accessibility problems based on what has been observed, user opinions, interviews, questionnaires and so on.

An objective evaluation of web accessibility requires more than the simple use of automatic tools for the evaluation of websites [23]. Effective evaluations are conducted with end users and web accessibility experts.

3 Methodology

An SLR makes a synthesis of the information published in different scientific databases in an orderly, precise and analytical way about a particular issue. The SLR methodology aims to guide the review process on a topic of interest to determine its research progress and to find new areas of research. This research adopts the Kitchenham’s guidelines, which include the following steps: “Planning Review, Conducting Review and Reporting Review” [24]. Figure 4 presents the flowchart of the SLR methodology used to determine accessibility findings and trends on university websites.

Fig. 4

Flowchart of the SLR methodology

3.1 Planning the systematic literature review

The aim of this stage is to determine the need for an SLR and to establish a review protocol. To determine the need for an SLR, an exhaustive search of SLRs in different scientific databases is performed.

3.1.1 Identification of the need for a systematic literature review

Using the following publications [24,25,26,27] as a reference, a search string was created to find similar SLRs on the accessibility of university websites and to determine whether the proposed SLR of this study will contribute to filling any gaps. Two equivalent search strings were created, one for the Web of Science database and one for Scopus database:

• Web of Science: TI=((“web accessibility” OR accessibilit* OR WCAG) AND (universit* OR “higher education” OR education*) AND (web* OR portal) AND (“systematic literature review” OR “literature review” OR “systematic review”));

• Scopus: TITLE((“web accessibility” OR accessibilit* OR WCAG) AND (universit* OR “higher education” OR education*) AND (web* OR portal) AND (“systematic literature review” OR “literature review” OR “systematic review”)).

The search with these strings yielded two literature reviews that are summarized and analyzed below:

• 1. In 2017, an SLR [28] was conducted to identify web accessibility issues on the websites of Saudi universities and governments. The search was conducted on different databases from 2009 to 2017, analyzing a total of 123 articles. In the results, the authors revealed that web accessibility is a universal problem and that many countries in the world, among them Saudi Arabia, face web accessibility issues.

• 2. In 2020, an SLR [29] was conducted on the empirical results of the evaluation of accessibility of educational websites. The search was conducted on different databases from January 2009 to October 2019, analyzing a total of 25 papers. This SLR raised 10 research questions. The first three questions carried out the bibliometric analysis of the selected papers, and the remaining seven questions carried out the literature review. The authors in the results determined that 80 % of the analyzed studies make use of automatic evaluation tools, 8 % real users and 12 % a combination of experts, automatic tools and real users. The authors concluded that the educational websites analyzed in the SLR have accessibility problems.

In summary, the first SLR identifies accessibility problems of websites of Saudi governments and universities. The second SLR analyzes the empirical results of the evaluation of accessibility of educational websites. Our SLR presented in this paper, unlike the other two, aims to determine the process of evaluating the accessibility of university websites worldwide. To this end, we start by identifying the continents and countries where accessibility research has been carried out on university websites (RQ1 [RQ1.1, RQ1.2]). Then, the screening process applied for the selection of the websites or other resources that have been evaluated is determined (RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3]). The accessibility standards, accessibility laws, versions of the WCAG and levels of conformance used in the evaluations are defined (RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3]). The methods, tools, end users or experts that help in the evaluation are defined (RQ7, RQ8, RQ9). The most common errors found in the results and their compliance with the levels of accessibility of the university websites are analyzed in the chosen studies (RQ10, RQ11).

3.1.2 Development of a review protocol

This research aims to compile papers published up to March 2021 on the accessibility of university websites in three scientific databases. To this end, a review protocol is developed, defining the research questions, the search strategy, the inclusion and exclusion criteria and quality assessment of the selected papers, which are analyzed and detailed below.

Research questions

The research questions are classified into four main groups:

• 1. What was evaluated?

• 2. What standards were used for the evaluation?

• 3. What methods were used for the evaluation?

• 4. What results were obtained?

Eleven research questions were formulated, some with several sub-questions to achieve the objective of this SLR.

• The research questions and sub-questions, objectives and expected results formulated in Table 1 will answer the first research question (RQ1 [RQ1.1, RQ1.2], RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3]);

• The research questions and sub-questions, objectives and expected results formulated in Table 2 will answer the second research question (RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3]);

• The research questions, objectives and expected results formulated in Table 3 will answer the third research question (RQ7, RQ8, RQ9);

• The research questions, objectives and expected results formulated in Table 4 will answer the fourth research question (RQ10, RQ11). These questions can be answered only if the accessibility evaluation considered the WCAG.

Table 1 Research questions related to the coverage of the evaluated universities—What was evaluated?

Table 2 Research questions related to the standards, laws and methods applied in the selection—What standards were used for the evaluation?

Table 3 Research questions related to the methods, tools and types of users who have evaluated the accessibility—What methods were used for the evaluation?

Table 4 Research questions related to accessibility errors and conformance levels—What results were obtained?

The scope in an investigation allows to delimit the causes of the facts or phenomena being studied. Once the research questions and sub-questions were defined, the scope of the research was determined using the PICOC method proposed by Petticrew and Roberts [30]:

• P: Population. Published papers on web accessibility;

• I: Intervention. University websites;

• C: Comparison. Comparison between accessibility standards, accessibility evaluation methods and the methods used for the selection of university websites to be evaluated;

• O: Outcomes. Generate social responsibility in the administrators and developers of university websites;

• C: Context. University environments.

In the results of this SLR, we answer the research questions and sub-questions defined in Tables 1, 2, 3, 4, through the analysis, synthesis and interpretation of the results found in the selected papers. The main contribution of this SLR is the consolidation of the results of the selected papers to determine findings and trends in the subject matter under investigation.

Search strategy

A search string is a key piece in an SLR for item selection, as it delimits the scope and coverage of the investigation. The keywords and their replacement (substitution) terms for the search scope have been determined according to the research questions and sub-questions defined in Tables 1, 2, 3, 4. These keywords and their replacement terms used in the search scope of this SLR can be seen in Table 5.

Table 5 Search scope

The Boolean operators used in the search string are OR to join the replacement terms, so that the results show at least one of these terms, AND to combine the main parts of the search string where the results show all the search terms. These Boolean operators have been combined several times to create a custom search string. Moreover, the wildcard (*) has been applied to represent both the plural and singular of each keyword or replacement term and also to search for keywords containing certain characters. Finally, double quotes have been used to search for exact phrases. For the search in the scientific databases Web of Science, Scopus and IEEE Xplore, we use a specific search string for each database. The specific search strings used in each scientific database are listed below:

• Web of Science: ((TI=“web accessibility” OR TI= accessibilit*) AND (TI=universit* OR TI=“higher education” OR TI= education*) AND (AB=website* OR AB= “web site” OR AB=“web sites” OR AB=web OR AB= portal) AND (AB=eval* OR AB=anal*));

• Scopus: (TITLE ((“web accessibility” OR accessibilit*)) AND TITLE((universit* OR “higher education” OR education)) AND TITLE-ABS ((eval* OR anal*)) AND TITLE-ABS((website* OR “web site” OR “web sites” OR web OR portal)));

• IEEE Xplore: ((“Document Title” :“web accessibility” OR “Document Title” :accessibilit*) AND (“Document Title” :universit* OR “Document Title” :“higher education” OR “Document Title” :education*) AND (“Abstract” :website* OR “Abstract” :“web site” OR “Abstract” :“web sites” OR “Abstract” :web OR “Abstract” :portal) AND (“Abstract” :eval* OR “Abstract” :anal*)).

The title, the abstract and the keywords describe in a concrete way what will be found in the content of a paper. For this reason, three specific search strings have been created to search the keywords and replacement terms defined in Table 5, in the titles and abstracts of the articles published in the scientific databases.

Inclusion and exclusion criteria

The selection process of the studies plays a very important role in the results of the SLR. Therefore, all studies found with the search strings were evaluated to determine whether they should be included in this research. Papers that did not meet all inclusion criteria were excluded from the review. Papers that met at least one of the exclusion criteria were excluded. The year of publication of the papers was not taken into account as a parameter of inclusion and exclusion. The inclusion criteria used in this SLR are presented below:

• I1. Papers published in a journal AND;

• I2. Papers written in English AND;

• I3. Papers that at least mention the accessibility of university websites in the abstract.

The exclusion criterion was used to discard papers that met the following conditions:

• E1. Papers that are secondary research (e.g., an SLR) OR;

• E2. Duplicated papers OR;

• E3. Papers that are not research papers (e.g., letter to the editor, erratum).

Quality assessment

Quality assessment (QA) is intended to contribute to the selection of papers through a set of questions that must be answered to guide the research [24]. The indexing of the journals in which the works have been published (Scimago Journal & Country Rank (SJR) and Journal Citation Reports (JCR)) has also been incorporated into the quality questions. Seven questions have been defined to measure the quality of each paper. Each question has a score of 1; therefore, the maximum total score is 7. Table 6 shows the QA questions that were applied to each paper.

Table 6 Quality assessment checklist [31]

3.2 Conducting the systematic literature review

3.2.1 Identification of research

An SLR allows the analysis of literature on specific research topics. To develop an SLR, it is necessary to determine the search terms and to define the scientific databases where the search will be carried out. In a study conducted in 2019 [32], the search quality of PubMed, Google Scholar and other 26 academic search databases was evaluated; the results showed that Google Scholar is not suitable as a primary search resource. For this reason, the most notable scientific databases in the field of research were selected for this SLR, such as Web of Science, Scopus and IEEE Xplore Digital Library. We estimate that these databases are sufficient because there are even duplicate papers in the search results, i.e., the same paper appears in several databases at the same time, which shows that the coverage of these scientific databases is very high. This collection of databases of bibliographic references was selected using the following criteria:

• They collect references from scientific publications that disseminate scientific knowledge;

• They index high-quality papers that are peer-reviewed;

• They allow customized searches using query operators.

3.2.2 Selection of studies

The selection process covered papers indexed in the scientific databases until March 2021. In the Scopus scientific database 104 papers were found, in Web of Science 99 papers and in the IEEE Xplore 16 papers. Of the 219 papers, the selection was made applying the inclusion and exclusion criteria as follows: 98 papers were discarded because they were not published in a journal (I1); 35 papers were discarded because they were not written in the English language (I2); 19 papers were discarded because the abstract did not describe university websites (I3); 2 papers were discarded because they were the result of an SLR (E1); 23 papers were discarded because they were duplicated (E2), and finally, no papers were discarded because they were all research papers (E3). In summary, of the 219 papers found, 177 were discarded, leaving 42 papers selected for this SLR. The flowchart of the study selection can be seen in Fig. 5.

Fig. 5

Flowchart of studies selection

3.2.3 Quality assessment

To achieve the objective of the SLR, papers must comply with the quality assessment parameters [33] defined in Table 6. After evaluating the papers, the sum of the results obtained from each paper was made. Then, the values obtained were normalized to standardize the results. To this end, the minimum–maximum normalization formula [34] was used, which calculates the final values on a scale of 0 to 1. Below is the formula used to calculate this value (1):

$$\begin{aligned} Normalization = \frac{Score - min(Score)}{[max(Score) - min(Score)]} \end{aligned}$$

(1)

where the min(Score) has a value of 0, the max(Score) has a value of 7, and the Score takes the value of the sum of the QA values of each paper. Papers with a normalized score of less than 0.70 were excluded from the SLR.

3.3 Reporting the systematic literature review

This stage aims to answer the research questions and sub-questions presented in Tables 1, 2, 3, 4. The following section presents the literature review answering questions RQ1 [RQ1.1, RQ1.2], RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3], RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3], RQ7, RQ8, RQ9, RQ10 and RQ11, making a summary and synthesis of the data collected from the results of the selected studies.

4 Results

In this section, we first present the results of the QA of the selected papers. Secondly, we answer each research question by summarizing and discussing the results of the selected papers.

Table 7 shows the 42 papers yielded by the search and selection process. Each paper includes the reference, the publication year and the name of the journal where it was published. In addition, the QA results for each one of the selected papers can also be seen in Table 7. This table is sorted by publication year. To standardize the sum of the QA values, a normalization column was created in which the final compliance value of each paper is calculated between 0 and 1. After the calculation, it was possible to see that several items scored 0.71, but were included in the SLR. These articles were not published in journals indexed in SJR (QA6) and JCR (QA7).

Table 7 Results of the quality assessment of the selected papers, sorted by publication year

4.1 Systematic literature review

4.1.1 RQ1. Where has the accessibility of the universities been analyzed the most?

To answer this research question, two research sub-questions have been answered. These questions identify the continents and countries where university websites were analyzed in the selected papers.

RQ1.1 In which continents has the accessibility of university websites been analyzed the most?

According to the results, we can see that 47.6 % of the selected papers analyze the accessibility of university websites in Asia, 33.3 % in America, 9.5 % in Europe and 2.4 % in Africa. However, there are 3 papers evaluating university websites on more than one continent (6.12 %). The number of papers per continent or continents can be seen in Fig. 6 (see Table 15, in “Appendix” A, for full data).

Fig. 6

Number of papers per continent

RQ1.2 In which countries has the accessibility of university websites been analyzed the most?

A map has been developed to visually present the scope of web accessibility evaluation in the world (see Fig. 7). The largest number of countries per continent that have evaluated university websites is:

• America: 7,865 universities in 28 countries;

• Asia: 1,003 universities in 18 countries;

• Europe: 235 universities in 18 countries;

• Africa: 31 universities in 2 countries;

• Oceania: 6 universities in 2 countries.

Fig. 7

Map of the universities evaluated by country and continent

A total of 9,140 university websites worldwide have been evaluated. In summary, the scope of the evaluation is 5 continents and 68 countries. In Fig. 8, we can see the number of university websites evaluated by country and continent (see Table 15, in “Appendix” A, for full data).

Fig. 8

Number of university websites evaluated per country and continent (There is a gap in the scale for the sake of clarity)

4.1.2 RQ2. How have been the universities selected for analysis and how many universities have been analyzed?

To answer this research question, two research sub-questions have been posed. They analyze the number of universities and the methods used for the selection of the web pages of the universities evaluated in the selected papers.

RQ2.1 What methods have been used for the selection of university websites to be evaluated?

The mode of selection of the universities has been classified into three types: sampling, randomly selected and all universities. When the authors refer to all universities, they are considering those universities that belong to a certain classification or country. In the 42 papers analyzed, 83.33 % of the papers selected universities by sampling, 11.90 % by random selection and 4.77 % all universities. The results can be seen in Fig. 9 (see Table 16, in “Appendix” A, for full data).

Fig. 9

Methods used for the selection of the websites of the universities to be evaluated

The comparison between the WCAG and the methods used in the selection of university websites is briefly presented below:

• WCAG 1.0. The selection of university websites for web accessibility evaluation was made using the following methods: sampling (11 papers), randomly selected (1 paper), all universities (1 paper);

• WCAG 2.0. The methods for selecting university websites for web accessibility evaluation are as follows: sampling (21 papers), randomly selected (3 papers), all universities (1 paper);

• WCAG 2.1. The selection of university websites for web accessibility evaluation was made using the following methods: sampling (3 papers), randomly selected (1 paper).

As a result of the comparison, the trend in the use of “sampling” for the selection of university websites in the three versions of the WCAG is evident. However, there are also significantly fewer who use “randomly selected” to determine the university websites to be evaluated.

RQ2.2 How many university websites have been evaluated?

A total of 9,140 university websites have been evaluated in the 42 selected papers. The paper with the highest number of universities evaluated is [45], which analyzes 3,251 universities. The average number of universities evaluated worldwide is 217.62. In Table 8, we can see the descriptive statistical summary of the universities evaluated per paper (see Table 16, in “Appendix” A, for full data).

Table 8 Descriptive statistical summary of the university websites evaluated per paper

4.1.3 RQ3 What type of pages, how many and what other resources have been evaluated on university websites?

To answer this research question, three research sub-questions have been posed. They analyze the types of web pages, how many web pages and what other resources have been evaluated on the websites of the universities of the selected papers.

RQ3.1 What type of web pages have been evaluated?

Of the 42 papers analyzed, 37 papers evaluate the home pages of the university websites. The papers [47, 48, 54, 76] do not refer in their content to the web pages that have been evaluated in the universities. The papers [37] and [63] evaluated the library homepage. The results are presented in Fig. 10 (see Table 16, in “Appendix” A, for full data).

Fig. 10

Types of web pages evaluated at universities

RQ3.2 How many web pages have been evaluated?

A total of 38,416 web pages have been evaluated in the 42 selected papers. The paper with the highest number of evaluated web pages is [45], which analyzes 31,701 web pages. The accessibility data of the web pages were collected through a semi-automatic procedure developed in PHP and using the Google Custom Search API (Google). In Table 9, we can see the descriptive statistical summary of the web pages evaluated per paper (see Table 16, in “Appendix” A, for full data).

Table 9 Descriptive statistical summary of the web pages evaluated per paper

RQ3.3 What other resources have been evaluated?

Other resources, such as videos and PDF documents, have been evaluated on university websites. A total of 91,421 YouTube videos have been evaluated in the paper [76] and 28,395 PDF documents in the paper [45] (see Table 16, in “Appendix” A, for full data).

4.1.4 RQ4. What are the web accessibility standards used to assess university websites?

The standards used in evaluating the accessibility of university websites are ISO/IEC 40500:2012 [6], Section 508 [77] and a combination of the two standards. Section 508 and WCAG 1.0 were used for the evaluation of university websites in 8 papers, ISO/IEC 40500:2012 in 22 papers and the combination of the two standards ISO/IEC 40500:2012 and Section 508 in 3 papers. WCAG 1.0 is used for the evaluation of university websites in 5 papers and WCAG 2.1 in 4 papers. Figure 11 shows the results (see Table 15, in “Appendix” A, for full data).

Fig. 11

Number of papers by web accessibility standard used in the evaluation of university websites

WCAG 2.1 has been used in papers [66, 72, 73, 76] to evaluate the accessibility of university websites. Taking into account that WCAG 2.1 adopts all the principles, guidelines and success criteria of WCAG 2.0, we could say that these papers also use ISO/IEC 40500:2012. Therefore, from this point of view, the total number of papers that use ISO/IEC 40500:2012 is 29 and this is the web accessibility standard most used in the evaluations.

4.1.5 RQ5. What are the accessibility laws mentioned?

The web accessibility laws and policies [78] have been taken as a reference to answer this question, although this is not an exhaustive or definitive list. After reviewing the 42 selected papers, the articles that make use of an accessibility law are cited:

• [63]—“SI 5568, Israeli web accessibility guidelines” ;

• [73]—“Stanca Act, Italian accessibility legislation.”

Therefore, only 4.76 % of the analyzed papers make use of a web accessibility law in their evaluation. The remaining 40 papers do not mention any web accessibility law (see Table 15, in “Appendix” A, for full data).

4.1.6 RQ6. How are the WCAG used to assess university websites?

To answer this research question, three research sub-questions have been answered. They determine the conformance levels and the versions of the WCAG used for the evaluation of university websites. In addition, it is verified whether the WCAG-EM methodology is used in the evaluation of the websites of the universities analyzed in the selected papers.

RQ6.1 What are the WCAG versions used to assess university websites?

Considering the official versions of the WCAG published by the W3C, it has been determined that WCAG 1.0 has been used to evaluate university websites in 13 papers, WCAG 2.0 in 25 papers and WCAG 2.1 in 4 papers. Figure 12 shows the trend in the use of the different versions of WCAG over time (see Table 17, in “Appendix” A, for full data).

Fig. 12

Number of papers per version of the WCAG and year of publication

RQ6.2 What are the conformance levels used to assess university websites?

Of the 42 selected papers, all make use of one of the three versions of WCAG. In 10 papers, university websites are evaluated with conformance level A, 5 papers with conformance level AA and 1 paper with conformance level AAA. In addition, the websites of 5 papers were evaluated with A and AA conformance levels and 12 papers with A, AA and AAA conformance levels. It should be noted that 9 papers [35, 36, 39, 40, 42, 43, 45, 46, 57] do not stipulate the conformance levels used for the evaluation of university websites. The papers by conformance levels can be seen in Fig. 13 (see Table 17, in “Appendix” A, for full data).

Fig. 13

Number of papers by conformance levels

RQ6.3 Is WCAG-EM used to assess university websites?

All analyzed papers use some of the steps of the WCAG-EM Methodology; however, none of them stipulates the whole use of this methodology in their manuscript. Therefore, none of the papers makes a strict use of the WCAG-EM methodology in the evaluation of university websites (see Table 17, in “Appendix” A, for full data).

4.1.7 RQ7. What are the methods used to assess university websites?

Taking into account the web accessibility evaluation methods defined in Section II Background, they have been classified into three types: 1) automatic evaluation tools (35 papers), 2) manual evaluation (end users and experts, 4 papers) and 3) manual and automatic evaluation (combination of the two methods, 3 papers). Figure 14 presents the evaluation methods used in the selected papers (see Table 18, in “Appendix” A, for full data).

Fig. 14

Evaluation methods used in the selected papers

4.1.8 RQ8. What are the automatic tools used to assess university websites?

Considering the web accessibility evaluation tools published on the W3C website [20], it has been determined that 38 selected papers use automatic evaluation tools in their evaluation. Therefore, 90.47 % of university websites are evaluated using automatic tools. Figure 15 shows the number of papers per automatic evaluation tool (see Table 18, in “Appendix” A, for full data).

Fig. 15

Number of papers by automatic evaluation tools

The automatic tools most used in the evaluation of university websites are: AChecker (15 papers), WAVE (14 papers), Bobby (9 papers) and TAW (7 papers). By making an analysis, it is determined that:

• Bobby. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 1.0 in papers published between 2002 and 2013;

• AChecker. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 1.0 and WCAG 2.0 in papers published between 2011 and 2020;

• TAW. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 2.0 in papers published between 2014 and 2019;

• WAVE. This tool was used to automatically evaluate the accessibility of university websites with WCAG 2.0 and WCAG 2.1 in papers published between 2015 and 2020.

Bobby, launched in 1996 [79], is one of the first tools for web accessibility evaluation, which had several versions one more advanced than another and lost its continuity in 2005. The other accessibility evaluation tools have adapted to the changes in WCAG. In summary, the most widely used tool with WCAG 1.0 for automatic evaluation of university websites is Bobby. With the publication of WCAG 2.0, university websites were evaluated with the automatic tools AChecker, TAW and WAVE. Finally, with the publication of WCAG 2.1, the automatic tool that trends in use over time is WAVE. Figure 16 shows the trend of Bobby, AChecker, TAW and WAVE over time.

Fig. 16

Trend of the most widely used automatic evaluation tools in the evaluation of the accessibility of university websites over time

4.1.9 RQ9. Which experts and users helped assess university websites?

The papers that have used experts or real users for the evaluation of university websites are the following (see Table 18, in “Appendix” A, for full data):

• One paper [37] presented an automatic evaluation using Bobby and a manual evaluation using JAWS, Internet Explorer and the help of two experts from university websites;

• One paper [42] presented a manual evaluation of university websites using the Internet Explorer 7.0, Web Accessibility Toolbar (WAT), JAWS and the expert knowledge of the authors of the document;

• One paper [54] presented an automatic evaluation using CynthiaSays and a manual evaluation with 16 blind users of the websites of the analyzed universities;

• One paper [68] presented a heuristic review of university websites using the UX Check Tool and a manual evaluation with two experts in web accessibility;

• One paper [76] presented the manual evaluation of the YouTube videos published on university websites with the expert knowledge of the authors of the paper on web accessibility.

4.1.10 RQ10. What are the main errors found on university websites?

The main accessibility errors by checkpoint and priority were identified in the papers that perform the evaluation with WCAG 1.0. The most common errors found by checkpoint in the selected papers are: lack of text alternatives (1.1), missing language (4.3), tables used for layout or with complex structure (5.5) and poor navigation (12.1, 13.1). The highest percentage of errors is in Priority 1 with 40 %, Priority 2 with 35 % and Priority 3 with 25 %. The errors by priority, checkpoint and number of papers can be seen in Table 10 (see Table 19, in “Appendix” A, for full data).

Table 10 Errors by priority, checkpoints and number of papers WCAG 1.0

In the papers that perform the evaluation with WCAG 2.0, the most common errors have been identified by principle and success criteria. The percentage of errors by principle was Operable with 36 %, Perceivable with 32 %, Understandable with 24 % and Robust with 8 %. The errors by principle, success criteria, conformance levels and number of papers can be seen in Table 11 (see Table 19, in “Appendix” A, for full data).

Table 11 Errors by principle, conformance level, success criteria and number of papers WCAG 2.0

According to the results, the errors have been classified by guideline. These errors are presented in the content of the websites as follows:

• Guideline 1.1—Text alternatives Non-text content has no alternative text;

• Guideline 1.3—Adaptable The content is not presented in different ways on the website without losing its structure and information;

• Guideline 1.4—Distinguishable The content cannot be seen and heard;

• Guideline 2.1—Keyboard accessibility Websites do not allow interaction using the keyboard;

• Guideline 2.4—Navigable The website does not include help that allows the user to find the content;

• Guideline 3.1—Readable The content of the websites is not clear, concrete and concise;

• Guideline 3.2—Predictable Websites do not function in a predictable manner;

• Guideline 3.3—Input Assistance The websites do not have a guide for filling in the information on the forms;

• Guideline 4.1—Compatible The websites do not allow the use of assistive technology tools.

In the papers that perform the evaluation with WCAG 2.1, the most common errors have been identified by principle and success criteria. The percentage of errors by principle was: Perceivable with 56 %, Operable with 28 %, Understandable with 13 % and Robust with 3 %. The most common errors classified by guideline are:

• Guideline 1.1—Text alternatives Non-text content has no alternative text;

• Guideline 1.2—Time-based Media There are no time-based alternatives in the media;

• Guideline 1.3—Adaptable The content is not presented in different ways on the website without losing its structure and information;

• Guideline 1.4—Distinguishable The content cannot be seen and heard;

• Guideline 2.1—Keyboard accessibility Websites do not allow interaction using the keyboard;

• Guideline 2.2—Enough Time Users are not given enough time to use and read the content;

• Guideline 2.4—Navigable The website does not include help that allows the user to find the content;

• Guideline 3.2—Predictable Websites do not function in a predictable manner;

• Guideline 3.3—Input Assistance The websites do not have a guide for filling in the information on the forms;

• Guideline 4.1—Compatible The websites do not allow the use of assistive technology tools.

The errors by principle, success criteria, conformity levels and number of papers can be seen in Table 12 (see Table 19, in “Appendix” A, for full data):

Table 12 Errors by principle, conformance level, success criteria and number of papers WCAG 2.1

4.1.11 RQ11. What is the accessibility compliant status of university websites?

The websites of 9,140 universities have been evaluated on the selected papers using automatic and manual methods. However, none of the websites of the universities that were analyzed were found to be compliant with web accessibility standards or with the WCAG (see Table 19, in “Appendix” A, for full data).

5 Discussion

The discussion is divided into two parts. In the first part, a bibliometric analysis of the selected papers is made. In the second part, we discuss the results found in the analyzed papers.

The bibliometric analysis begins by determining the trend of publication of research results on the evaluation of the accessibility of university websites over time. As important data, an increase has been observed in recent years, from 2016 to 2020, in the research of web accessibility. This trend can be seen in Fig. 17 (see Table 14, in “Appendix” A, for full data).

Fig. 17

Number of papers published per year

The papers analyzed in this SLR have been published in 27 journals. In Tables 7 or 14, we can consult the name and the acronym of each journal. The UAIS is the journal with the largest number of papers, which is considered normal, since the UAIS “addresses the accessibility, usability, and, ultimately, acceptability of Information Society Technologies by anyone, anywhere, at anytime, and through any media and device” [80]. Figure 18 shows the number of papers published per journal (see Table 14, in “Appendix” A, for full data).

Fig. 18

Number of papers published per journal

The indexation of the journals where the papers have been published is detailed below (see Table 14, in “Appendix” A, for full data):

• 18 papers in SJR and JCR;

• 20 papers in SJR;

• 4 papers are not indexed in SJR and JCR.

Figure 19 shows graphically the number of papers published in journals indexed in SJR and JCR. The ranking quartiles of the journals indexed in SJR and JCR were determined by the year of publication of the papers. Papers published in 2020 were assigned the 2019 quartiles. Journals that do not have a quartile have been placed in the figure as QX (see Table 14, in “Appendix” A, for full data).

Fig. 19

Number of papers published in SJR and JCR

After the data were extracted from the selected papers, they were analyzed and interpreted in the results of this SLR. With these data, it has been possible to answer the following questions:

• 1. What was evaluated?

  In answering this question, we determined that the web pages evaluated on the university websites are: university homepage, search page, list of university colleges, departments, and/or degree programs, campus directory of faculty, admissions homepage, staff and/or students, course listings, employment homepage, academic calendar, job listings, campus map and library homepage. In addition, PDF documents and YouTube videos published on university websites have been evaluated.

• 2. What standards were used for the evaluation?

  The web accessibility standards used in the evaluation of university websites are: WCAG 1.0, WCAG 2.0 (ISO/IEC 40500:2012), WCAG 2.1 and Section 508.

• 3. What methods were used for the evaluation?

  The evaluation methods used in the selected papers are: manual evaluation, evaluation with automatic tools and the combination of both methods.

• 4. What results were obtained?

  The results showed that the web pages, PDF documents and YouTube videos evaluated in the selected papers were not accessible when they were evaluated.

Considering that the selected papers come from different points in time, a time trend of their results has been made. Of the 42 selected papers, 11 were conducted in the USA. Therefore, we have made a timeline of the accessibility results in US university websites (shown in Fig. 20), analyzing the results over time. It can be seen that accessibility problems persist over time. This may be due to the new versions of the WCAG, which are becoming more and more comprehensive and evolving over time. In addition, it is quite difficult to compare the results from different studies because there does not exist a baseline and each study applies different methods and tools.

Fig. 20

Results over time of the evaluation of university websites in the USA

We also analyzed the results over time of the evaluation of university websites in countries with more than one article among those selected. Table 13 presents the results. In these papers, accessibility problems also persist over time.

Table 13 Results over time of the evaluation of university websites

The website of a university is the most common introduction for many people interested in the university. This is why universities publish their academic offers, regulations, university projection, achievements, among other things on their websites, in order to make their services and their work as a university known to the community. However, for university websites to reach a wider audience, they must comply with web accessibility laws and regulations. Unfortunately, according to the results there seems to be a lack of knowledge or interest in the laws that should be applied, because in many countries policies and laws on web accessibility already exist [78].

Universities have always been at the forefront of technological change. However, according to the papers analyzed, it can be seen that there are isolated efforts in some universities to incorporate the WCAG into their websites.

Also, as an interesting fact, we have been able to verify that only papers [45, 76] have dealt with the evaluation of other resources that exist on the web pages, and the rest have only focused on the web pages. In order for university websites to comply with WCAG, accessibility laws or standards, their web pages and their content must be accessible.

It should be noted that university websites do not appear to be created with web accessibility in mind. This may be due to the lack of knowledge of the website developers. However, universities have a duty to make their websites accessible, to provide universal access to their content. Article 9 of the CRPD [1] stipulates that governments should “promote access for persons with disabilities to new information and communications technologies and systems, including the Internet” and “promote the design, development, production and distribution of accessible information and communications technologies and systems at an early stage” . To this end, it is necessary that the developers of university websites are trained and apply the WCAG in their web pages and in the content that is published.

6 Limitations of the study

An SLR can be affected by a series of limitations. One of these is bias in data collection by the authors. For this reason, this SLR starts by defining customized search strings with the key terms of this research. These search strings were applied in the extraction of data from each of the scientific databases. In addition, to reduce this bias, we have applied inclusion, exclusion and quality criteria in the selection of the papers. It is worth mentioning that all the authors were involved in the whole SLR process to give more strength to their results.

Another limitation found is that the authors of some of the selected papers did not contrast the errors found with the WCAG success criteria. Therefore, the authors of this SLR contrasted these errors with the WCAG success criteria under their knowledge. In addition, some papers do not detail the errors found in the evaluation.

Also, this SLR does not include papers not published in journals such as conferences, editorials, prefaces, discussion, comments, tutorial summaries, workshop summaries and panels. Therefore, “grey” literature is not considered in this SLR.

Another limitation is time, as WCAG has been evolving from WCAG 1.0 [14], WCAG 2.0 [15], WCAG 2.1 [7], to draft WCAG 2.2 [16] and first public working draft WCAG 3.0 [81], and each of them seeks a wider scope.

7 Conclusions and future work

The aim of this SLR was to examine the accessibility of university websites as reported in 42 selected papers. The 42 papers analyzed evaluate 9,140 universities distributed in 5 continents and 68 countries. In summary, the results obtained are the following:

• 83.33 % of the universities were selected through a sampling;

• 38,416 web pages, 91,421 YouTube Videos and 28,395 PDF documents were evaluated;

• 88.10 % of the papers only evaluated the homepage of the websites;

• The standards used were ISO/IEC40500:2012, Section 508 and the combination of both;

• The evaluation methods used were manual methods, methods with automatic tools and the combination of the two;

• 90.47 % of the university websites were evaluated with automatic tools;

• The most common errors are presented in the following guidelines: Text Alternatives, Adaptive, Distinguishable, Keyboard Accessibility, Navigable, Readable, Predictable, Input Assistance and Compatible.

The results allow us to conclude that university websites show some similar types of violations over time with the WCAG success criteria. Another important finding is that we see a change from WCAG 1.0 to 2.0 and then the recent appearance of 2.1, its evolution in the use for the evaluation of the accessibility of university websites in the world. Also, the W3C lists 154 tools [20] for the evaluation of web accessibility in an automatic way, while 21 tools have been used in the papers analyzed in this SLR. It can be observed that there is a great concentration in a few tools (AChecker, Bobby, TAW and WAVE) that are the preferred ones, although there are many more.

The global education system has been disrupted by the SARS-COV-2 pandemic. Most institutions had to adapt to online education when their facilities closed. Teachers, students and administrative staff had to go through this transition, but for many people with and without disabilities the situation became very difficult. For this reason, the technological tools for teaching–learning have been adapted according to the needs of the virtual world and the experience of teachers, without measuring their barriers. In order to reduce these barriers, it is necessary to evaluate the acceptance and benefits of technological tools before their application. In addition, the websites of educational institutions must comply with accessibility standards in their portals. Universal access on the web benefits people with and without disabilities.

The developers of university websites must take into account the concept of accessibility from the beginning when creating or redesigning a website. In addition, there are resources that can help achieve accessibility of a website such as templates, plugins, libraries and others. Also, websites in their development phase should be tested with real users and experts.

In addition, this work offers a review of accessibility on university websites that may be useful for others to understand this concept. In future work, a multivocal literature review [82] will be carried out on the accessibility of the contents published on university websites. In addition, it will be reviewed if there are software development architectures that implement accessibility in a website. Also, as future work, it is proposed to make a comparison of the accessibility of public and private university websites and the acceptance of Learning Management Systems (LMS) by people with disabilities.

Appendix: Data collected and errors

See Tables 14, 15, 16, 17, 18, 19, 20 and 21

The information contained in this section will help the reader understand the review process described in this document. In addition, the data extraction and accessibility errors found in the results of the selected papers are detailed.

Table 14 Bibliometric data extraction

Table 15 Data collected for RQ1, RQ4, RQ5

Table 16 Data collected for RQ2 and RQ3

Table 17 Data collected for RQ6

Table 18 Data collected for RQ7, RQ8, RQ9

Table 19 Data collected for RQ10 and RQ11

Table 20 Errors per priorities and checkpoints WCAG 1.0

Table 21 Errors per principles, guidelines and success criteria WCAG 2.1