Skip to main content
SpringerLink
Log in

Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

American sign language recognition is still a research focus in computer vision community. Recently, most researches mainly extract low-level features for hand gesture recognition. These approaches perform poorly on recognizing gestures posed like a fist. In this paper, we propose a novel multimodal framework for sign language recognition system which exploits the Leap Motion Controller (LMC) and a webcam. We compute two sets of features. The first set is the angles at hand joints acquired by the LMC sensor. When, hand poses like a fist, the positions of the thumb joints captured by the LMC are not very precise. So, we should incorporate the second set of features extracted from the hand shape contour provided by a webcam. In this paper, we introduce a new mid-level feature, called Contour Segment Code (CSC), to represent hand shape contour. The proposed shape representation, first, extracts meaningful landmarks from the hand shape contour. CSC then encodes different segments of the hand contour into a code based on the shape landmarks. The extracted landmarks precisely determine the hand direction. The proposed method is tested by creating a very challenging dataset composed of 64,000 samples. Our experiments study the performance of the LMC and characteristics of CSC in different scenarios. The experimental results demonstrate the privileged performance of the proposed method against the systems which use depth images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

References

  1. Amit Y, Geman D (1994) Randomized Inquiries About Shape: An Application to Handwritten Digit Recognition, CHICAGO UNIV IL DEPT OF STATISTICS

  2. Amit Y, Geman DJNC (1997) Shape quantization and recognition with randomized trees. Neural Comput 9(7):1545–1588

    Article  Google Scholar 

  3. Avola D, Bernardi M, Cinque L, Foresti GL, Massaroni C (2019) Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Trans Multimed 21(1):234–245

    Article  Google Scholar 

  4. Behera SK, Dogra DP, Roy PPJESWA (2018) Fast recognition and verification of 3D air signatures using convex hulls. Expert Syst Appl 100:106–119

    Article  Google Scholar 

  5. Behera SK, et al. (2018) Analysis of 3D signatures recorded using leap motion sensor. multimedia Tools and Applications, 1–26

  6. Bendib MM, Merouani HF, Diaba F (2015) Automatic segmentation of brain MRI through stationary wavelet transform and random forests. Pattern Anal Applic 18(4):829–843

    Article  MathSciNet  Google Scholar 

  7. Bernardos AM, Sánchez JM, Portillo JI, Wang X, Besada JA, Casar JR (2016) Design and deployment of a contactless hand-shape identification system for smart spaces. J Ambient Intell Humaniz Comput 7(3):357–370

    Article  Google Scholar 

  8. Breiman L (2017) Classification and regression trees. Routledge

  9. Cao J, Yu S, Liu H, Li P (2016) Hand posture recognition based on heterogeneous features fusion of multiple kernels learning. Multimed Tools Appl 75(19):11909–11928

    Article  Google Scholar 

  10. Chen M, Liu C (2018) And G.J.I.S.R. Du, A human–robot interface for mobile manipulator. Intell Serv Robot 11(3):269–278

    Article  Google Scholar 

  11. Criminisi A, Shotton J (2013) Decision forests for computer vision and medical image analysis. Springer Science & Business Media

  12. Cui J, Sourin AJC, Graphics (2018) Mid-air interaction with optical tracking for 3D modeling. Comput Graph 74:1–11

    Article  Google Scholar 

  13. Dardas NH, Georganas NDJITOI, measurement (2011) Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans Instrum Meas 60(11):3592–3607

    Article  Google Scholar 

  14. Feng B, He F, Wang X, Wu Y, Wang H, Yi S, Liu W (2017) Depth-projection-map-based bag of contour fragments for robust hand gesture recognition. Trans Human-Mach Syst 47(4):511–523

    Article  Google Scholar 

  15. Hisham B, Hamouda AJJ (2017) Arabic static and dynamic gestures recognition using leap motion. J Comput Sci 13(8):337–354

    Article  Google Scholar 

  16. Hu K, L Yin (2013) Multi-scale topological features for hand posture representation and analysis. in Proceedings of the IEEE International Conference on Computer Vision

  17. Huang D-Y, W-C Hu, S.-H. Chang (2009) Vision-based hand gesture recognition using PCA+ Gabor filters and SVM. In 2009 fifth international conference on intelligent information hiding and multimedia signal processing. IEEE

  18. Ibrahim M (2017) Scalability and performance of random forest based learning-to-rank for information retrieval. In ACM SIGIR Forum. ACM

  19. Just A, Y Rodriguez, S Marcel (2006) Hand posture classification and recognition using the modified census transform. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06). IEEE

  20. Kausar N, Majid AJPA (2016) And applications, Random forest-based scheme using feature and decision levels information for multi-focus image fusion. Pattern Anal Applic 19(1):221–236

    Article  Google Scholar 

  21. Keskin C et al. (2012) Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In European Conference on Computer Vision. Springer

  22. Kim, M., J.Y.J.M.T. Lee, and Applications, Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality. Multimed Tools Appl, 2016. 75(23): p. 16529–16550.

  23. Kirac F, Kara YE, Akarun LJPRL (2014) Hierarchically constrained 3D hand pose estimation using regression forests from single frame depth data. Pattern Recogn Lett 50:91–100

    Article  Google Scholar 

  24. Kumar P et al (2016) Study of text segmentation and recognition using leap motion sensor. IEEE Sensors J 17(5):1293–1301

    Article  Google Scholar 

  25. Kumar P, Saini R, Roy PP, Dogra DP (2017) 3D text segmentation and recognition using leap motion. Multimed Tools Appl 76(15):16491–16510

    Article  Google Scholar 

  26. Kumar P, Saini R, Roy PP, Pal U (2018) A lexicon-free approach for 3D handwriting recognition using classifier combination. Pattern Recogn Lett 103:1–7

    Article  Google Scholar 

  27. Kumar P, Roy PP, Dogra DPJIS (2018) Independent bayesian classifier combination based sign language recognition using facial expression. Inf Sci 428:30–48

    Article  MathSciNet  Google Scholar 

  28. Li N, Cheng X, Guo H, Wu Z (2016) Recognizing human interactions by genetic algorithm-based random forest spatio-temporal correlation. Pattern Anal Applic 19(1):267–282

    Article  MathSciNet  Google Scholar 

  29. Marin G, Dominio F, Zanuttigh P (2016) Hand gesture recognition with jointly calibrated leap motion and depth sensor. Multimed Tools Appl 75(22):14991–15015

    Article  Google Scholar 

  30. Nai W, Liu Y, Rempel D, Wang Y (2017) Fast hand posture classification using depth features extracted from random line segments. Pattern Recogn 65:1–10

    Article  Google Scholar 

  31. Pedersoli F, Benini S, Adami N, Leonardi R (2014) XKin: an open source framework for hand pose and gesture recognition using kinect. Vis Comput 30(10):1107–1122

    Article  Google Scholar 

  32. Priyal, S.P. And P.K.J.P.R. Bora, A robust static hand gesture recognition system using geometry based normalizations and Krawtchouk moments. Pattern Recogn, 2013. 46(8): p. 2202–2219.

    Article  Google Scholar 

  33. Pugeault N, R Bowden (2011) Spelling it out: Real-time ASL fingerspelling recognition. in IEEE International conference on computer vision workshops (ICCV workshops)

  34. Rastgoo R, Kiani K, Escalera SJE (2018) Multi-modal deep hand sign language recognition in still images using restricted Boltzmann machine. Entropy 20(11):809

    Article  Google Scholar 

  35. Ren Y, Xie X, Li G, Wang Z (2018) Hand gesture recognition with multiscale weighted histogram of contour direction normalization for wearable applications. IEEE Trans Circ Syst Video Technol 28(2):364–377

    Article  Google Scholar 

  36. Ristin-Kaufmann M (2015) Large-Scale Image Recognition with Random Forests. Doctoral dissertation, ETH Zurich

  37. Rivera-Acosta M, Ortega-Cisneros S, Rivera J, Sandoval-Ibarra F (2017) American sign language alphabet recognition using a Neuromorphic sensor and an artificial neural network. Sensors 17(10):2176

    Article  Google Scholar 

  38. Schapire REJML (1990) The strength of weak learnability. Mach Learn 5(2):197–227

    Google Scholar 

  39. Tang D (2015) 3D hand pose regression with variants of decision forests, Imperial College London

  40. Togootogtokh E, Shih TK, Kumara WGCW, Wu SJ, Sun SW, Chang HH (2018) 3D finger tracking and recognition image processing for real-time music playing with depth sensors. Multimed Tools Appl 77(8):9233–9248

    Article  Google Scholar 

  41. Triesch J, von der Malsburg CJI, Computing V (2002) Classification of hand postures against complex backgrounds using elastic graph matching. Image Vis Comput 20(13–14):937–943

    Article  Google Scholar 

  42. Wang, C., Z. Liu, And S.-C.J.I.T.O.M. Chan, Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed, 2015. 17(1): p. 29–39.

    Article  Google Scholar 

  43. Yue-long C, Ling C, Gen-cai CJJOZU (2013) Hierarchical Bag-of-Features with Spectral-HIK filter based hand posture recognition. J ZheJiang Univ (Eng Sci) 47(9):1531–1536

    Google Scholar 

  44. Zeng W, Wang C, Wang Q (2018) Hand gesture recognition using leap motion via deterministic learning. Multimed Tools Appl 77:28185–28206

    Article  Google Scholar 

  45. Zhang H, Li H, Zhou MJJOHU (2013) Hand posture recognition based on multi-feature and compressive sensing. J Hunan Univ (Nrl Sci) 40(3):87–92

    Google Scholar 

  46. Zhang C, Tian YJCV, Understanding I (2015) Histogram of 3D facets: a depth descriptor for human action and hand gesture recognition. Comput Vis Image Underst 139:29–39

    Article  Google Scholar 

  47. Zhou Y, Jiang G, Lin YJPR (2016) A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recogn 49:102–114

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Vision Research Lab, Department of Electrical Engineering, Sahand University of Technology, Tabriz, Iran

    Khadijeh Mahdikhanlou & Hossein Ebrahimnezhad

Authors
  1. Khadijeh Mahdikhanlou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossein Ebrahimnezhad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hossein Ebrahimnezhad.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahdikhanlou, K., Ebrahimnezhad, H. Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding. Multimed Tools Appl 79, 22235–22259 (2020). https://doi.org/10.1007/s11042-020-08982-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08982-8

Keywords

Search

Navigation