Abstract
Background/Introduction
The deaf community in the world uses a gesture-based language, generally known as sign language. Every country has a different sign language; for instance, USA has American Sign Language (ASL) and UK has British Sign Language (BSL). The deaf community in Pakistan uses Pakistan Sign Language (PSL), which like other natural languages, has a vocabulary, sentence structure, and word order. Majority of the hearing community is not aware of PSL due to which there exists a huge communication gap between the two groups. Similarly, deaf persons are unable to read text written in English and Urdu. Hence, the provision of an effective translation model can support the cognitive capability of the deaf community to interpret natural language materials available on the Internet and in other useful resources.
Methods
This research involves exploiting natural language processing (NLP) techniques to support the deaf community by proposing a novel machine translation model that translates English sentences into equivalent Pakistan Sign Language (PSL). Though a large number of machine translation systems have been successfully implemented for natural to natural language translations, natural to sign language machine translation is a relatively new area of research. State-of-the-art works in natural to sign language translation are mostly domain specific and suffer from low accuracy scores. Major reasons are specialised language structures for sign languages, and lack of annotated corpora to facilitate development of more generalisable machine translation systems. To this end, a grammar-based machine translation model is proposed to translate sentences written in English language into equivalent PSL sentences. To the best of our knowledge, this is a first effort to translate any natural language to PSL using core NLP techniques. The proposed approach involves a structured process to investigate the linguistic structure of PSL and formulate the grammatical structure of PSL sentences. These rules are then formalised into a context-free grammar, which, in turn, can be efficiently implemented as a parsing module for translation and validation of target PSL sentences. The whole concept is implemented as a software system, comprising the NLP pipeline and an external service to render the avatar-based video of translated words, in order to compensate the cognitive hearing deficit of deaf people.
Results and Conclusion
The accuracy of the proposed translation model has been evaluated manually and automatically. Quantitative results reveal a very promising Bilingual Evaluation Understudy (BLEU) score of 0.78. Subjective evaluations demonstrate that the system can compensate for the cognitive hearing deficit of end users through the system output expressed as a readily interpretable avatar. Comparative analysis shows that our proposed system works well for simple sentences but struggles to translate compound and compound complex sentences correctly, which warrants future ongoing research.
Similar content being viewed by others
References
Sofiane Boucenna, Antonio Narzisi, Elodie Tilmont, Filippo Muratori, Giovanni Pioggia, David Cohen, Mohamed Chetouani (2014) Interactive technologies for autistic children: a review, cognitive computation, Volume 6, Number 4, Page 722
Khan NS, et al. Speak Pakistan: challenges in developing Pakistan Sign Language using information technology. South Asian Studies. 2015;30(2):367.
Luqman H, Mahmoud SA. Transform-based Arabic sign language recognition. Procedia Computer Science. 2017;117:2–9.
Nießen, S., Och, F.J., Leusch, G., Ney, H., Informatik, L.F.: An evaluation tool for machine translation: fast evaluation for MT research. In: In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000) (2000).
Abid, K., Khan, N. S., Farooq, U., Farooq, M. S., Naeem, M. A., & Abid, A. (2018). A roadmap to elevate Pakistan Sign Language among regional sign languages. South Asian Studies (1026-678X), 33(2).
Mohandes M, Deriche M, Liu J. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE transactions on human-machine systems. 2014;44(4):551–7.
Zhao L, et al. A machine translation system from English to American Sign Language. Conference of the Association for Machine Translation in the Americas. Berlin, Heidelberg: Springer; 2000.
Bonham, M.E.: English to ASL Gloss Machine Translation. M. Art thesis, Brigham Young University (2015).
Wang P, Song Q, Han H, Cheng J. Sequentially supervised long short-term memory for gesture recognition. Cogn Comput. 2016;8(5):982.
Kröger BJ, Birkholz P, Kannampuzha J, Kaufmann E, Mittelberg I. Movements and holds in fluent sentence production of American Sign Language: the action-based approach. Cogn Comput. 2011;3(3):449.
Marshall, Ian, and Éva Sáfár. A prototype text to British Sign Language (BSL) translation system. Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 2. Association for Computational Linguistics, 2003.
San Segundo Hernández, R., Lopez Ludeña, V., Martin Maganto, R., Sánchez, D., & García, A. (2010). language resources for Spanish-Spanish Sign Language (LSE) translation.
Porta J, López-Colino F, Tejedor J, Colás J. A rule-based translation from written Spanish to Spanish Sign Language glosses. Comput Speech Lang. 2014;28(3):788–811.
Van Zijl, Lynette, and Andries Combrink. The South African sign language machine translation project: issues on non-manual sign generation. Proceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries. South African Institute for Computer Scientists and Information Technologists, 2006.
Shoaib, Umar, et al. Integrating multiwordnet with Italian sign language lexical resources. Expert Systems with Applications41.5 (2014): 2300-2308.
Bungeroth, J., & Ney, H. (2004). Statistical sign language translation. In Workshop on representation and processing of sign languages, LREC (Vol. 4, pp. 105-108).
Cox S, Lincoln M, Tryggvason J, Nakisa M, Wells M, Tutt M, et al. The development and evaluation of a speech-to- sign translation system to assist transactions. Int J Hum Comput Interact. 2003;16(2):141–61.
Veale T, Conway A, Collins B. The challenges of cross-modal translation: English-to-sign-language translation in the zardoz system. Mach Transl. 1998;13(1):81–106.
d’Armond, L.S.: Representation of American sign language for machine translation, Ph.D. thesis, Georgetown University (2002).
Marshall, I., Sáfár, É.: Extraction of semantic representations from syntactic SMU link grammar linkages. In: Proceedings of Recent Advances in Natural Language Processing, pp. 154–159 (2001)
San-Segundo R, Montero JM, Macías-Guarasa J, Córdoba R, Ferreiros J, Pardo JM. Proposing a speech to gesture translation architecture for Spanish deaf people. J Vis Lang Comput. 2008;19(5):523–38.
Othman A, Jemni M. Statistical sign language machine translation: from English written text to American sign language gloss. Int J Comput Sci Issues. 2011;8(5):65–73.
Kouremenos D, Ntalianis K, Kollias S. A novel rule based machine translation scheme from Greek to Greek Sign Language: production of different types of large corpora and Language Models evaluation. Comput Speech Lang. 2018;51:110–35.
Luqman H, Mahmoud SA. Automatic translation of Arabic text-to-Arabic sign language. Univ Access Inf Soc. 2018:1–13.
Nguyen TBD, Phung TN, Vu TT. A rule-based method for text shortening in Vietnamese Sign Language translation. In Information systems design and intelligent applications. Singapore: Springer; 2018. p. 655–62.
Verma, V. K., & Srivastava, S. (2018). Toward machine translation linguistic issues of Indian Sign Language. In Speech and language processing for human-machine communications (pp. 129-135). Springer, Singapore
Yasir, F., Prasad, P. W. C., Alsadoon, A., Elchouemi, A., & Sreedharan, S. (2017). Bangla Sign Language recognition using convolutional neural network. In Intelligent computing, instrumentation and control technologies (ICICICT), 2017 International Conference on (pp. 49-53). IEEE.
Tumsri, J., & Kimpan, W. (2017). Thai sign language translation using leap motion controller. In Proceedings of The International MultiConference of Engineers and Computer Scientists 2017 (pp. 46-51).
Zeshan, U. (2000). Sign language in Indo-Pakistan: a description of a signed language. John Benjamins Publishing.
Zeshan U. Indo-Pakistani Sign Language grammar: a typological outline. Sign Language Studies. 2003;3:157–212.
Abbas, A., & Sarfraz, S. (2018). Developing a prototype to translate text and speech to Pakistan Sign Language with bilingual subtitles: a framework. Journal of Educational Technology Systems
Khan N, Shahzada A, Ata S, Abid A, Khan Y, ShoaibFarooq M. A vision based approach for Pakistan Sign Language alphabets recognition. Pensee. 2014;76(3).
Hassan B, Farooq MS, Abid A, Sabir N. Pakistan Sign Language: computer vision analysis & recommendations. VFAST Transactions on Software Engineering. 2015;9(1):1–6.
Othman A, Jemni M. Designing high accuracy statistical machine translation for sign language using parallel corpus: case study English and American Sign Language. Journal of Information Technology Research (JITR). 2019;12(2):134–58.
Stoll, Stephanie, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. Text2Sign: towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision (2020): 1-18.
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., ... & Vogler, C. (2019). Sign language recognition, generation, and translation: an interdisciplinary perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (pp. 16-31).
Shaalan K. Rule-based approach in Arabic natural language processing. The International Journal on Information and Communication Technologies (IJICT). 2010;3(3):11–9.
Filhol M, Hadjadj MN, Testu B. A rule triggering system for automatic text-to-sign translation. Univ Access Inf Soc. 2016;15(4):487–98.
Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology-volume 1 (pp. 173-180). Association for Computational Linguistics .
Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project
De Marneffe, M. C., & Manning, C. D. (2008). Stanford typed dependencies manual (pp. 338-345). Technical report, Stanford University.
Hadla LS, Hailat TM, Al-Kabi MN. Evaluating Arabic to English machine translation. Int J Adv Comput Sci Appl (IJACSA). 2014;5(11):68–73.
Gonàlez, M., Giménez, J., Màrquez, L.: A graphical interface for MT evaluation and error analysis. In: The 50th Annual Meeting of the Association for Computational Linguistics (2012).
Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational lin- guistics, Association for Computational Linguistics, pp. 311–318 (2002).
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: Proceedings of association for machine translation in the Ameri- cas, vol. 200 (2006).
Mahmud M, Kaiser MS, Hussain A, Vassanelli S. Applications of deep learning and reinforcement learning to biological data. IEEE Transactions in Neural Networks and Learning Systems. 2018;29(6):2063–79.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Khan, N.S., Abid, A. & Abid, K. A Novel Natural Language Processing (NLP)–Based Machine Translation Model for English to Pakistan Sign Language Translation. Cogn Comput 12, 748–765 (2020). https://doi.org/10.1007/s12559-020-09731-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-020-09731-7