1 Introduction

According to the World Health Organization (WHO) [27], there are around 466 million people worldwide with disabling hearing loss. Of these, 34 million are children, and schools catering to their needs are very few in number. It is also seen [11, 25, 27] that the unemployment rate among deaf adults is very high, and they are in the lower grades of employment when compared with the general workforce. According to Census 2011 [7], India has 5 million people with hearing disability. This is about 18.9% of the total disabled population.

In a typical classroom environment, teachers without proper knowledge and training in Sign Language (SL) will use oral communication which hinder the learning process of the deaf students. Studies have shown that deaf children lag behind hearing children in vocabulary development [22]. This will affect their academic performance. In India, deaf students follow the same curriculum as their hearing peers. Because of the lack of widely accepted writing forms, deaf children have to learn to read in a second language [23]. In the context of the Indian state of Kerala, the official language is Malayalam. As per the Higher secondary curriculum, students have to learn Malayalam and Hindi as second and third languages and Science and History is taught in English medium.

There is very little emphasis on sign language-based instructions in India, which makes it arduous for deaf children in language comprehension. Studies have shown that providing bilingual instructions in sign language helps in learning [5, 6, 9, 12, 17, 20, 28]. SignText is built on this principle.

A tool like SignText is important for two major reasons. Firstly, difficulty in following spoken language instructions limits the learning of deaf children both at school and home. In addition to this, most of the deaf children are born to hearing parents. They may not be well versed in sign language and will not favour it, thus making it difficult for the deaf children to learn using sign language. Also, most of the teachers have spoken language ability and may not show interest in teaching in sign language. Moreover, a publicly available sign-based learning tool will help parents, social workers and researchers to learn and understand sign language without any external help. This is very important since lack of learning tools is a hindrance to their learning.

Representing sign-based content in printed media has obvious limitations, since it is challenging to represent gestures as images. This favours the use of multimedia-based solutions as gestures can be easily integrated in it. Instructions through virtual mode are experimented in many parts of the world due to the COVID-19 pandemic. This unexpected switch proved to be a challenge for both teachers and students. This is more serious in developing countries [15] since the level of technology adoption is much less as compared to the developed world. SignText application enables learning through virtual mode. Since it provides sign and text-based instructions, students can follow textbook contents without the help of an instructor. This is confirmed in the study conducted. SignText, proposed in this paper is a multimedia-based tool for learning instructions in sign language. The following features are also provided by the SignText application:

  • Ability to start playing videos from user defined point;

  • Ability to pause and play video based on the user need;

  • Highlighting available words for which corresponding signs are available;

  • Searching for a word and displaying the occurrences;

To measure the effectiveness of this tool, a detailed study was conducted, and the following research questions were considered:

  • RQ1: How good is SignText-based learning as compared to normal textbook-based learning ?

  • RQ2: Do students like features provided by the SignText application ?

  • RQ3: What is the usability of the SignText application ?

2 Related works

Availability of easily accessible sign language-based learning tools for textbook contents is minimal [14]. Most of the existing tools are multi-modal bilingual applications and are game-based [4, 8]. The authors have not come across any of them for Indian Sign Language (ISL). Many Automatic Sign Language Recognition (ASLR) [29] and Virtual reality [1,2,3]-based applications are also reported in literature. Though tools that augment textbook contents with Sign Language-based instructions are scarce in the literature, there exist some tools that help in improving literacy skills through Sign Language-based instructions for deaf learners.

The RISE ebook [10] is a bilingual bimodal joint project by Gallaudet University and Swarthmore College to promote reading. The RISE ebook makes available classic children’s books and is augmented with Sign Language to provide a shared reading facility for deaf children and their parents. Currently, ebooks are available in American, Korean, Nepali, Fiji, Japanese and Brazilian sign languages.

Veliza, Espinoza et al. [26] studied the effect of SL augmented digital books for deaf learners. They followed a participatory development model and preliminary results indicate that augmenting SL-based instructions with books improves comprehension. The Visual Language and Visual Learning (VL2) project from Gallaudet University developed a storybook application for mobile phones and tablets. The VL2 project [21] augments SL contents with English text, facilitating reading and language acquisition for early and emerging readers. Unlike the above two projects, the SMARTSign project [24] is a mobile-based application used along with the physical copy of books. In this application, user can click a picture of the story, and the SMARTSign AR mobile application automatically converts it to the corresponding sign. Though not specific to textbook learning, Hariharan, Al-hkhazraji, and Huenerfauth [16] proposed a web-based tool to translate words to ASL for improving accessibility. Initial results show that participants preferred having support tools in their interfaces as opposed to none.

3 Design and implementation of SignText

The SignText interface is designed to give importance to sign videos along with the text. An iterative method is followed for SignText development. After developing an initial prototype, evaluation is conducted with the help of an instructor and a student and based on the feedback received, the SignText application is modified.

SignText is designed to work as a web-based application. Along with providing easy accessibility, it offers a low-cost mechanism for access. The main screen of SignText is divided into two sections, one showing sign and the other showing the corresponding text. Video controls are placed on the left side of the web page where user can start/stop/pause the video. SignText can play a sentence corresponding to the currently playing frame. The sentence currently playing is highlighted for ease of access. This feature makes it easy for the user to understand which sentence is being played. Since a single page cannot hold the whole lesson contents, after the given text is finished, SignText automatically replaces the current text with the old one. This is accomplished without reloading a new page. This is another helpful accessibility feature provided in SignText. This has been confirmed by the end survey conducted on students. User can click on a sentence/word, and the system plays the corresponding sign language translation. If the sign is not available for the selected word, SignText will not play anything. SignText also displays links to each page where the user can click, and the corresponding page will be loaded. A search feature is provided in the page which allows the user to search for a word, and the sentences that contain the search key will be displayed.

SignText has two main components, Text view and a Video view. These two areas function synchronously, depending on whether the whole text is played or sentence/word is selected. These two views works synchronously based on the controls. Currently four controls are provided:

  1. 1.

    Play: Plays the contents from the current state;

  2. 2.

    Pause: Pause at the current state;

  3. 3.

    Stop: Stop and move to the initial state;

  4. 4.

    Seek: Move to the desired state.

Two additional controls are available in the displayed text:

  1. 1.

    Single mouse click: Selecting word or sentence;

  2. 2.

    Drag and select: Selecting a word or sentence.

Through the drag and select operation, more than one sentence can be selected. All these controls can be applied at any time during the working of SignText.

The server side of the software uses a database to store video and text and an index file to store information about video position corresponding to the entire text contents. When the user selects a text, only the corresponding video and synchronisation information are moved to the client. The client side has a buffer to store video, text and an index file to store details about video position corresponding in the text in the cache. This enables SignText to work without network connectivity.

3.1 SignText content creation

SignText study materials are created with the help of an instructor of a school where the study is conducted. India has its own sign language ISL, but there are many regional variations. SignText study materials are created with the help of an instructor from the school where the study is conducted. This made sure that students are familiar with the signs of the instructor. A summary of the content of a chapter was created with the help of this instructor, and the content was later converted to sign language, line by line. After finalising the content, the videos are recorded for each line. These videos are saved separately and joined afterwards. The correctness of the content is verified by the instructor subsequently. The text and videos are then inserted into the web page template.

A pilot run was performed with the help of a student to ascertain whether he understood the signs. This student was excluded from further studies.

3.2 Implementation

SignText is implemented as a web-based application. Other than a web browser, no plugins are needed for running the application in it. This makes it easy to run SignText in almost all platforms. SignText is implemented using client-side scripting language. It offers many advantages to the end-user. Some of the features provided by SignText are the following:

  • SignText can immediately interact with the user without having to send a request to the server;

  • No Internet connection is needed for SignText to work;

  • SignText can work on almost all modern browsers;

  • SignText does not need any extra plugins to work;

  • SignText can work independently of Operating System.

Since JavaScript is used for providing functionalities, it enables users to work without having a web server, making it easy to work on most systems. Videos are handled using HTML5. This made it possible to play video without any external plugin. Since most web browsers currently support HTML5, SignText can easily be run on them. Videos are stored in H.264 MP4 format and is supported by most browsers including Mozilla Firefox, Google Chrome and Safari.

The flow of control of SignText is in the following manner. When the user launches the web browser, it displays the contents of the page, including text and video. The user starts the application SignText by clicking on the start button. This event starts the Javascript engine and plays the sign video, simultaneously highlighting the current line in the text. The user can “pause” and “stop” the video using the buttons whenever he/she wants. The pause and stop functions are handled by the Javascript engine.

4 Experimental design

A total of 34 grade-12 students from the St.Clare Oral Higher Secondary School For The Deaf, Kerala participated in the experiment spanning two weeks. In Kerala, there are 60,925 people with hearing disability which amounted to 7.67% of the total disabled population in the state as per the 2015 disability census report [19]. The number of participants is limited since the number of students attending school for the deaf is much less as compared to the hearing peers. Participant details are given in Table 1. Mean age of the participants is 17.62.

Table 1 Participant details
Full size table

The subject selected for the study is Malayalam, which is the official language of the state of Kerala, the place where the study was conducted. The lesson selected for the study was regarding social reformers. Students were willing to participate in the study because it gave them an opportunity to learn the concept through sign language.

The sample was divided into two equal groups: experimental and control. Students were randomly selected and put in two groups. The experimental group used SignText for learning, and the control group used the textbook for learning. Each group was individually evaluated after the classroom session and after three days of using their assigned learning method. An exam was conducted to assess the performance of the students.

A pretest, post-test research design was followed for the study. For both tests, multiple-choice questions were asked to find out the user understanding of the topics discussed. For the present study, a lesson previously discussed in class was considered. For the pretest, students were given 20 multiple choice questions from a topic that was already taught in class. The test was administered by the teacher, post which three days were given to both groups to study the topics. To ensure that the same amount of time was given to both the groups for study, the textbooks were kept at school, and the students were not given prior information about the post-test. For the post-test, 20 multiple choice questions were administered to both groups, and scores were calculated. Both groups were administered the same set of questions for pre and post-tests. Overview of the research design is shown in Figure 1.

Fig. 1
figure 1

Research design

Full size image

In addition, a usability study was conducted on the experimental group to find the usability of SignText application. For the usability study, a System Usability Scale (SUS) [13, 18] survey was conducted, and the score was calculated. The questionnaire for the usability study involved ten questions which can be answered in a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree) scale. SUS is popular because of its ease of administration and scoring. Scores for SUS are in the range 0-100 with a higher value indicating better-perceived usability. The following questions were asked for the study:

  1. 1.

    I think that I would like to use this system frequently;

  2. 2.

    I found the system unnecessarily complex;

  3. 3.

    I thought the system was easy to use;

  4. 4.

    I think that I would need the support of a technical person to be able to use this system;

  5. 5.

    I found the various functions in this system were well-integrated;

  6. 6.

    I thought there was too much inconsistency in this system;

  7. 7.

    I would imagine that most people would learn to use this system very quickly;

  8. 8.

    I found the system very cumbersome to use;

  9. 9.

    I felt very confident using the system;

  10. 10.

    I had to learn many things before I could get going with this system.

A qualitative study was also administered to get feedback from users regarding the SignText application. Open-ended questions were asked regarding the SignText application for this purpose.

5 Results and discussion

Figure 2 shows the SignText interface. Sign video is placed on the left side and text is placed on the right side. Controls are placed at the bottom portion of the interface. “Stop” and “Play/Pause” controls are provided. User can also use built-in controls of the video player. When the video is played, the corresponding text is highlighted with a light blue colour. The highlighted text is automatically moved as the playing progresses, for easy navigation.

Fig. 2
figure 2

SignText Screen

Full size image

Figure 3 shows students using SignText application.

Fig. 3
figure 3

Students using SignText application

Full size image

Pre- and post-test analysis results are given in Table 2 show that there is improvement in the mean scores of both groups upon completion of the study. Unlike other spoken languages, SL practised in India has no writing system of its own. In the present study, respondents were asked to write answers in either English or Malayalam, and the instructor helped them in writing down the answers.

Table 2 Pre-test post-test result statistics
Full size table

A null hypothesis that the scores of both groups would be equal and an alternative hypothesis that the score of the experimental group would be significantly higher than that of the control group were made before the test. Before running the t-test, normality of the scores was verified using Shapiro-Wilk normality test.

A paired one-tailed t-test was done to understand the effectiveness of the study for both the experimental (p-value = 0.0002951) group and the control group (p-value = 0.005144). Results show that the null hypothesis could be rejected. This confirms that learning (classroom and application learning) improves vocabulary understanding of both groups.

Gain scores for both groups were also calculated and compared. The study has found that there is a statistically significant difference between the gain score of the experimental group and the control group. This indicates that SignText application has an advantage in vocabulary learning. Table 3 shows the gain score test result for the vocabulary test. A one-tailed unpaired t-test is used here.

Table 3 Gain Score Summary
Full size table

Analysis of gain scores shows that the experimental group fares better than the control group in vocabulary learning.

The average SUS score obtained for the study is 87.5, which is considered good. This indicates that the perceived usability of SignText application is favourable. Since SUS survey questions included questions with positive and negative senses alternately and worded in English, it made it difficult for participants to understand the questions correctly. This is also reported in prior research [13]. Because of this, the instructor signed the questions for the participants and explained the significance of each question. Answers were also checked by the instructor and verified whether they answered based on the correct understanding. For the open-ended questions, participants favoured the bilingual instructions provided by SignText. Providing sign and fingerspelled version of the same was favoured by the participants. Some participants favoured putting the text below video rather than putting it in the top portion of the page. Participants suggested including signs corresponding to more words that are given in the text. This is important since such a feature provides more information regarding printed text in their own first language. Respondents expressed the view that it will improve language comprehension. Participants also expressed their desire to see contents like this for other subjects and languages. Respondents were of the opinion that getting to learn in their own language will greatly help in learning. The high SUS score obtained also supports this view. All the participants agreed that they were finding it difficult to follow the class when the instructor was not using sign language for communication. They also expressed the view that reading text and learning is difficult for them due to the delay in exposure to Sign Language. Since SignText provides sign-based instruction, it is helpful for them to learn at home in their language. Two students suggested including a summary of each content before its starting for ease of access.

All the participants agreed to the point that parallel placement of Sign videos and text helped in their learning. Also highlighting text with blue colour is found to be very effective. Participants who took part in the study were using sign augmented technology-based tool for learning for the first time. One respondent expressed the view that since SignText provides a feature to select a sentence and view the corresponding sign, it gives the opportunity to study, similar to a classroom environment where a sentence may be repeated to reiterate the concepts. All the respondents favoured the feature of selecting a sentence/word anywhere in the text to play its corresponding sign. They were of the opinion that the word selection feature would help in improving sign vocabulary. Three respondents suggested adding an image corresponding to the selected word in the sign videos for easy understanding.

Feedback from participants reveal that the SignText tool is helpful for them in learning textbook contents. SignText tool helps to learn textbook contents without the help of the instructor. This will enable students to revise the contents from home. This is important since lack of understanding of ISL by family members and unavailability of easily available multimedia sign dictionaries makes learning difficult for deaf students at home. As expressed by the participants, availability of contents signed by their own instructor makes it comfortable for the students in learning. Providing parallel text contents helps the students in understanding contents easily. Since SignText is provided as a web-based tool, it is easily accessible to students. This is reflected in the feedback and SUS scores. Stored contents makes it available for repeated viewing and thus helps in the learning process. All these prove that a better learning outcome can be achieved by using SignText tool.

SignText is designed with ease of use of content creation in mind. SignText helps instructors to easily record videos and add text corresponding to it. The instructor can also export the content easily for distribution. This helps in the content creation process without the knowledge of programming. This is important since instructors are not experts in using technology-based tools for content creation process.

6 Conclusion and future scope

This work presents SignText, a web-based tool for learning textbook lessons in sign language. Results indicate that sign language-based learning improves the students’ score as compared to normal textbook-based learning. SignText currently supports sign language-based learning for a few lessons. Sign videos are pre-recorded by the instructor and fed into the web page. During qualitative analysis, respondents favoured watching their instructor sign for them. More study should be conducted on the effectiveness of signs produced by another signer in SignText application. The SignText application currently includes only one chapter for learning. Adding more content may not be possible for a sign instructor, and therefore, automating the content generation process will greatly help instructors and students. Developing SignText as a framework for content creation is worth looking into. Though SignText application works well on desktop systems, on mobile browsers, SignText contents are not properly aligned making it difficult for learners. Since mobile phone adoption in the developing world is increasing in a rapid pace, the ability to work well on mobile browsers is an advantage for better adoption of SignText application.