Skip to main content
SpringerLink
Log in

A Novel Natural Language Processing (NLP)–Based Machine Translation Model for English to Pakistan Sign Language Translation

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Background/Introduction

The deaf community in the world uses a gesture-based language, generally known as sign language. Every country has a different sign language; for instance, USA has American Sign Language (ASL) and UK has British Sign Language (BSL). The deaf community in Pakistan uses Pakistan Sign Language (PSL), which like other natural languages, has a vocabulary, sentence structure, and word order. Majority of the hearing community is not aware of PSL due to which there exists a huge communication gap between the two groups. Similarly, deaf persons are unable to read text written in English and Urdu. Hence, the provision of an effective translation model can support the cognitive capability of the deaf community to interpret natural language materials available on the Internet and in other useful resources.

Methods

This research involves exploiting natural language processing (NLP) techniques to support the deaf community by proposing a novel machine translation model that translates English sentences into equivalent Pakistan Sign Language (PSL). Though a large number of machine translation systems have been successfully implemented for natural to natural language translations, natural to sign language machine translation is a relatively new area of research. State-of-the-art works in natural to sign language translation are mostly domain specific and suffer from low accuracy scores. Major reasons are specialised language structures for sign languages, and lack of annotated corpora to facilitate development of more generalisable machine translation systems. To this end, a grammar-based machine translation model is proposed to translate sentences written in English language into equivalent PSL sentences. To the best of our knowledge, this is a first effort to translate any natural language to PSL using core NLP techniques. The proposed approach involves a structured process to investigate the linguistic structure of PSL and formulate the grammatical structure of PSL sentences. These rules are then formalised into a context-free grammar, which, in turn, can be efficiently implemented as a parsing module for translation and validation of target PSL sentences. The whole concept is implemented as a software system, comprising the NLP pipeline and an external service to render the avatar-based video of translated words, in order to compensate the cognitive hearing deficit of deaf people.

Results and Conclusion

The accuracy of the proposed translation model has been evaluated manually and automatically. Quantitative results reveal a very promising Bilingual Evaluation Understudy (BLEU) score of 0.78. Subjective evaluations demonstrate that the system can compensate for the cognitive hearing deficit of end users through the system output expressed as a readily interpretable avatar. Comparative analysis shows that our proposed system works well for simple sentences but struggles to translate compound and compound complex sentences correctly, which warrants future ongoing research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Sofiane Boucenna, Antonio Narzisi, Elodie Tilmont, Filippo Muratori, Giovanni Pioggia, David Cohen, Mohamed Chetouani (2014) Interactive technologies for autistic children: a review, cognitive computation, Volume 6, Number 4, Page 722

  2. Khan NS, et al. Speak Pakistan: challenges in developing Pakistan Sign Language using information technology. South Asian Studies. 2015;30(2):367.

    Google Scholar 

  3. Luqman H, Mahmoud SA. Transform-based Arabic sign language recognition. Procedia Computer Science. 2017;117:2–9.

    Article  Google Scholar 

  4. Nießen, S., Och, F.J., Leusch, G., Ney, H., Informatik, L.F.: An evaluation tool for machine translation: fast evaluation for MT research. In: In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000) (2000).

  5. Abid, K., Khan, N. S., Farooq, U., Farooq, M. S., Naeem, M. A., & Abid, A. (2018). A roadmap to elevate Pakistan Sign Language among regional sign languages. South Asian Studies (1026-678X), 33(2).

  6. Mohandes M, Deriche M, Liu J. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE transactions on human-machine systems. 2014;44(4):551–7.

    Article  Google Scholar 

  7. Zhao L, et al. A machine translation system from English to American Sign Language. Conference of the Association for Machine Translation in the Americas. Berlin, Heidelberg: Springer; 2000.

    Google Scholar 

  8. Bonham, M.E.: English to ASL Gloss Machine Translation. M. Art thesis, Brigham Young University (2015).

  9. Wang P, Song Q, Han H, Cheng J. Sequentially supervised long short-term memory for gesture recognition. Cogn Comput. 2016;8(5):982.

    Article  Google Scholar 

  10. Kröger BJ, Birkholz P, Kannampuzha J, Kaufmann E, Mittelberg I. Movements and holds in fluent sentence production of American Sign Language: the action-based approach. Cogn Comput. 2011;3(3):449.

    Article  Google Scholar 

  11. Marshall, Ian, and Éva Sáfár. A prototype text to British Sign Language (BSL) translation system. Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 2. Association for Computational Linguistics, 2003.

  12. San Segundo Hernández, R., Lopez Ludeña, V., Martin Maganto, R., Sánchez, D., & García, A. (2010). language resources for Spanish-Spanish Sign Language (LSE) translation.

  13. Porta J, López-Colino F, Tejedor J, Colás J. A rule-based translation from written Spanish to Spanish Sign Language glosses. Comput Speech Lang. 2014;28(3):788–811.

    Article  Google Scholar 

  14. Van Zijl, Lynette, and Andries Combrink. The South African sign language machine translation project: issues on non-manual sign generation. Proceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries. South African Institute for Computer Scientists and Information Technologists, 2006.

  15. Shoaib, Umar, et al. Integrating multiwordnet with Italian sign language lexical resources. Expert Systems with Applications41.5 (2014): 2300-2308.

  16. Bungeroth, J., & Ney, H. (2004). Statistical sign language translation. In Workshop on representation and processing of sign languages, LREC (Vol. 4, pp. 105-108).

  17. Cox S, Lincoln M, Tryggvason J, Nakisa M, Wells M, Tutt M, et al. The development and evaluation of a speech-to- sign translation system to assist transactions. Int J Hum Comput Interact. 2003;16(2):141–61.

    Article  Google Scholar 

  18. Veale T, Conway A, Collins B. The challenges of cross-modal translation: English-to-sign-language translation in the zardoz system. Mach Transl. 1998;13(1):81–106.

    Article  Google Scholar 

  19. d’Armond, L.S.: Representation of American sign language for machine translation, Ph.D. thesis, Georgetown University (2002).

  20. Marshall, I., Sáfár, É.: Extraction of semantic representations from syntactic SMU link grammar linkages. In: Proceedings of Recent Advances in Natural Language Processing, pp. 154–159 (2001)

  21. San-Segundo R, Montero JM, Macías-Guarasa J, Córdoba R, Ferreiros J, Pardo JM. Proposing a speech to gesture translation architecture for Spanish deaf people. J Vis Lang Comput. 2008;19(5):523–38.

    Article  Google Scholar 

  22. Othman A, Jemni M. Statistical sign language machine translation: from English written text to American sign language gloss. Int J Comput Sci Issues. 2011;8(5):65–73.

    Google Scholar 

  23. Kouremenos D, Ntalianis K, Kollias S. A novel rule based machine translation scheme from Greek to Greek Sign Language: production of different types of large corpora and Language Models evaluation. Comput Speech Lang. 2018;51:110–35.

    Article  Google Scholar 

  24. Luqman H, Mahmoud SA. Automatic translation of Arabic text-to-Arabic sign language. Univ Access Inf Soc. 2018:1–13.

  25. Nguyen TBD, Phung TN, Vu TT. A rule-based method for text shortening in Vietnamese Sign Language translation. In Information systems design and intelligent applications. Singapore: Springer; 2018. p. 655–62.

    Book  Google Scholar 

  26. Verma, V. K., & Srivastava, S. (2018). Toward machine translation linguistic issues of Indian Sign Language. In Speech and language processing for human-machine communications (pp. 129-135). Springer, Singapore

  27. Yasir, F., Prasad, P. W. C., Alsadoon, A., Elchouemi, A., & Sreedharan, S. (2017). Bangla Sign Language recognition using convolutional neural network. In Intelligent computing, instrumentation and control technologies (ICICICT), 2017 International Conference on (pp. 49-53). IEEE.

  28. Tumsri, J., & Kimpan, W. (2017). Thai sign language translation using leap motion controller. In Proceedings of The International MultiConference of Engineers and Computer Scientists 2017 (pp. 46-51).

  29. Zeshan, U. (2000). Sign language in Indo-Pakistan: a description of a signed language. John Benjamins Publishing.

  30. Zeshan U. Indo-Pakistani Sign Language grammar: a typological outline. Sign Language Studies. 2003;3:157–212.

    Article  Google Scholar 

  31. Abbas, A., & Sarfraz, S. (2018). Developing a prototype to translate text and speech to Pakistan Sign Language with bilingual subtitles: a framework. Journal of Educational Technology Systems

  32. Khan N, Shahzada A, Ata S, Abid A, Khan Y, ShoaibFarooq M. A vision based approach for Pakistan Sign Language alphabets recognition. Pensee. 2014;76(3).

  33. Hassan B, Farooq MS, Abid A, Sabir N. Pakistan Sign Language: computer vision analysis & recommendations. VFAST Transactions on Software Engineering. 2015;9(1):1–6.

    Article  Google Scholar 

  34. Othman A, Jemni M. Designing high accuracy statistical machine translation for sign language using parallel corpus: case study English and American Sign Language. Journal of Information Technology Research (JITR). 2019;12(2):134–58.

    Article  Google Scholar 

  35. Stoll, Stephanie, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. Text2Sign: towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision (2020): 1-18.

  36. Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., ... & Vogler, C. (2019). Sign language recognition, generation, and translation: an interdisciplinary perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (pp. 16-31).

  37. Shaalan K. Rule-based approach in Arabic natural language processing. The International Journal on Information and Communication Technologies (IJICT). 2010;3(3):11–9.

    Google Scholar 

  38. Filhol M, Hadjadj MN, Testu B. A rule triggering system for automatic text-to-sign translation. Univ Access Inf Soc. 2016;15(4):487–98.

    Article  Google Scholar 

  39. Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology-volume 1 (pp. 173-180). Association for Computational Linguistics .

  40. Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project

  41. De Marneffe, M. C., & Manning, C. D. (2008). Stanford typed dependencies manual (pp. 338-345). Technical report, Stanford University.

  42. Hadla LS, Hailat TM, Al-Kabi MN. Evaluating Arabic to English machine translation. Int J Adv Comput Sci Appl (IJACSA). 2014;5(11):68–73.

    Google Scholar 

  43. Gonàlez, M., Giménez, J., Màrquez, L.: A graphical interface for MT evaluation and error analysis. In: The 50th Annual Meeting of the Association for Computational Linguistics (2012).

  44. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational lin- guistics, Association for Computational Linguistics, pp. 311–318 (2002).

  45. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: Proceedings of association for machine translation in the Ameri- cas, vol. 200 (2006).

  46. Mahmud M, Kaiser MS, Hussain A, Vassanelli S. Applications of deep learning and reinforcement learning to biological data. IEEE Transactions in Neural Networks and Learning Systems. 2018;29(6):2063–79.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Management and Technology, Lahore, Pakistan

    Nabeel Sabir Khan & Adnan Abid

  2. University of the Punjab, Lahore, Pakistan

    Kamran Abid

Authors
  1. Nabeel Sabir Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adnan Abid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kamran Abid
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adnan Abid.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, N.S., Abid, A. & Abid, K. A Novel Natural Language Processing (NLP)–Based Machine Translation Model for English to Pakistan Sign Language Translation. Cogn Comput 12, 748–765 (2020). https://doi.org/10.1007/s12559-020-09731-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09731-7

Keywords

Search

Navigation