Skip to main content
SpringerLink
Log in

Real Time Air-Written Mathematical Expression Recognition for Children’s Enhanced Learning

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Air-writing is the process where, without the assistance of any handheld device, users use finger or hand gestures to write a character or words in free space. Due to its simple writing style, it has a great advantage over conventional pen-and-paper-based systems. However, because of the absence of any common delimiting criterion, non-uniform characters, and different writing styles, it is a difficult task. In this work, we propose an air written Mathematical expression recognition system using webcam video as input. We employed a new hand detection model that recognizes the writing hand and tracks the fingertip movement to collect the trajectories and then the convolutional neural network (CNN) is used as a recognizer. Through our model children can explore basic ME evaluation on their own without any instructor’s help and can gain more knowledge with minimal effort. Experiments were conducted on a combination of MNIST, ISI Air-Written English numerals, RTD along our own air-written Math operators datasets (MAIR). To evaluate the robustness of our model, we also tested our model on a group of children where they fed the input by writing in the air and the input data was captured using a system webcam. In both cases, we achieved promising results for digit recognition as well as ME evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. https://shahinur.com/en/rtd/.

  2. https://github.com/adildsw/ISI-Air.

  3. http://yann.lecun.com/exdb/mnist/.

References

  1. Alam MS, Kwon KC, Alam MA, Abbass MY, Imtiaz SM, Kim N (2020) Trajectory-based air-writing recognition using deep neural network and depth sensor. Sensors 20(2):376

    Article  Google Scholar 

  2. Alam MA Abbass MY Imtiaz SM Kim N Alam MS, Kwon KC (2020) Trajectory-based air-writing recognition using deep neural network and depth sensor. Sensors (Basel)

  3. Arsalan M, Santra A (2019) Character recognition in air-writing based on network of radars for human-machine interface. IEEE Sens J 19(19):8855–8864

    Article  Google Scholar 

  4. Behera SK, Dogra DP, Roy PP (2018) Fast recognition and verification of 3d air signatures using convex hulls. Expert Syst Appl 100:106–119

    Article  Google Scholar 

  5. Bochkovskiy A, Wang CY, Mark Liao H-Y (2020) Yolov4: optimal speed and accuracy of object detection. CoRR, abs/2004.10934

  6. Chen H, Ballal T, Muqaibel AH, Zhang X, Al-Naffouri TY (2020) Air writing via receiver array-based ultrasonic source localization. IEEE Trans Instrum Meas 69(10):8088–8101

    Google Scholar 

  7. Chen M, AlRegib G, Juang B (2016) Air-writing recognitionpart ii: detection and recognition of writing activity in continuous stream of motion data. IEEE Trans Hum Mach Syst 46(3):436–444

    Article  Google Scholar 

  8. Chen M, AlRegib G, Sohauang B (2016) Air-writing recognitionpart i: Modeling and recognition of characters, words, and connecting motions. IEEE Trans Hum Mach Syst 46(3):403–413

    Article  Google Scholar 

  9. Chiu LW, Hsieh JW, Lai CR, Chiang HR, Cheng SC, Fan KC (2018) Person authentication by air-writing using 3D sensor and time order stroke context: ICSM 2018, Toulon, France, pp 260–273

  10. Dash A, Sahu A, Shringi R, Gamboa J, Afzal MZ, Malik MI, Dengel A, Ahmed S (2017) Airscript - creating documents in air. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 01, pp 908–913

  11. De O, Deb P, Mukherjee S, Nandy S, Chakraborty T, Saha S (2016) Computer vision based framework for digit recognition by hand gesture analysis. In: IEEE 7th IEMCON, pp 1–5

  12. Deng X, Zhang Y, Yang S, Tan P, Chang L, Yuan Y, Wang H (2018) Joint hand detection and rotation estimation using cnn. IEEE Trans Image Process 27(4):1888–1900

    Article  MathSciNet  MATH  Google Scholar 

  13. Fu Z, Xu J, Zhu Z, Liu AX, Sun X (2019) Writing in the air with wifi signals for virtual reality devices. IEEE Trans Mob Comput 18(2):473–484

    Article  Google Scholar 

  14. Hazra S, Santra A (2018) Robust gesture recognition using millimetric-wave radar systems. IEEE Sens Lett 2(4):1–4

    Article  Google Scholar 

  15. Hou Y, Li Z, Wang P, Li W (2018) Skeleton optical spectra-based action recognition using convolutional neural networks. IEEE Trans Circuits Syst Video Technol 28(3):807–811

    Article  Google Scholar 

  16. Hu Y, Peng L, Tang Y (2014) On-line handwritten mathematical expression recognition method based on statistical and semantic analysis. In: 2014 11th IAPR international workshop on document analysis systems. pp 171–175

  17. Huang Y, Liu X, Zhang X, Jin L (2016) A pointing gesture based egocentric interaction system: dataset, approach and application. In: 2016 IEEE conference on computer vision and pattern recognition workshops (CVPRW). pp 370–377

  18. Kane L, Khanna P (2017) Vision-based mid-air unistroke character input using polar signatures. IEEE Trans Hum Mach Syst 47(6):1077–1088

    Article  Google Scholar 

  19. Khan AU, Borji A (2018) Analysis of hand segmentation in the wild. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. pp 4710–4719

  20. Kumar P, Saini R, Behera SK, Dogra DP, Roy PP (2017) Real-time recognition of sign language gestures and air-writing using leap motion. In: 2017 Fifteenth IAPR international conference on machine vision applications (MVA). pp 157–160

  21. Kumar P, Saini R, Roy PP, Dogra DP (2017) Study of text segmentation and recognition using leap motion sensor. IEEE Sens J 17(5):1293–1301

    Article  Google Scholar 

  22. Lai S, Jin L, Yang W (2017) Toward high-performance online hccr: a cnn approach with dropdistortion, path signature and spatial stochastic max-pooling. Pattern Recogn Lett 89:02

    Article  Google Scholar 

  23. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: single shot multibox detector. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision – ECCV 2016. Springer International Publishing, Cham, pp 21–37

    Chapter  Google Scholar 

  24. Mahmoudi MT, Mojtahedi S, Shams S (2017) Ar-based value-added visualization of infographic for enhancing learning performance. Comput Appl Eng Educ 25(6):1038–1052

    Article  Google Scholar 

  25. Misra S, Singha J, Laskar RH (2018) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ascii characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135

    Article  Google Scholar 

  26. Modanwal G, Sarawadekar K (2016) Towards hand gesture based writing support system for blinds. Pattern Recogn 57:50–60

    Article  Google Scholar 

  27. Mohammadi S, Maleki R (2020) Air-writing recognition system for persian numbers with a novel classifier. Visual Comput 36(5):1001–1015

    Article  Google Scholar 

  28. Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). pp 4207–4215

  29. Mukherjee S, Ahmed SA, Dogra DP, Kar S, Roy PP (2019) Fingertip detection and tracking for recognition of air-writing in videos. Expert Syst Appl 136:217–229

    Article  Google Scholar 

  30. Murata T, Shin J (2014) Hand gesture and character recognition based on kinect sensor. Int J Distrib Sens Netw 10(7):278460

    Article  Google Scholar 

  31. Oyedotun O, Khashman A (2016) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 28:04

    Google Scholar 

  32. Poularakis S, Katsavounidis I (2016) Low-complexity hand gesture recognition system for continuous streams of digits and letters. IEEE Trans Cybern 46(9):2094–2108

    Article  Google Scholar 

  33. Qu C, Zhang D, Tian J (2015) Online kinect handwritten digit recognition based on dynamic time warping and support vector machine. J Inf Comput Sci 12(1):413–422

    Article  Google Scholar 

  34. Rahim M, Shin J, Islam M (2020) Hand gesture recognition-based non-touch character writing system on a virtual keyboard. Multimedia Tools Appl 79(17):11813–11836

    Article  Google Scholar 

  35. Rahim MA, Shin J, Islam MR (2018) Human-machine interaction based on hand gesture recognition using skeleton information of kinect sensor. In: Proc of ICAIT. ACM, p 7579

  36. Rahman A, Roy P, Pal U (2019) Continuous motion numeral recognition using rnn architecture in air-writing environment. Pattern recognition. Springer, Cham, pp 76–90

    Google Scholar 

  37. Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. CoRR, abs/1804.02767

  38. Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  39. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Cortes C, Lawrence N, Lee D, Sugiyama M, Garnett R (eds.), Advances in neural information processing systems. vol. 28. Curran Associates, Inc.

  40. Roy K, Mohanty A, Sahay RR (2017) Deep learning based hand detection in cluttered environment using skin segmentation. In: 2017 IEEE international conference on computer vision workshops (ICCVW). pp 640–649

  41. Roy P, Ghosh S, Pal U (2018) A cnn based framework for unistroke numeral recognition in air-writing. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR). pp 404–409

  42. Setiawan A, Pulungan R (2018) Deep belief networks for recognizing handwriting captured by leap motion controller. Int J Electr Comput Eng 8:46934704

    Google Scholar 

  43. Shin J, Kim CM (2017) Non-touch character input system based on hand tapping gestures using kinect sensor. IEEE Access 5:10496–10505

    Article  Google Scholar 

  44. Smith KA, Csech C, Murdoch D, Shaker G (2018) Gesture recognition using mm-wave sensor for human-car interface. IEEE Sens Lett 2(2):1–4

    Article  Google Scholar 

  45. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(56):1929–1958

    MathSciNet  MATH  Google Scholar 

  46. Tang D, Chang HJ, Tejani A, Kim T (2017) Latent regression forest: structured estimation of 3d hand poses. IEEE Trans Pattern Anal Mach Intell 39(7):1374–1387

    Article  Google Scholar 

  47. Wang P, Li W, Gao Z, Zhang J, Tang C, Ogunbona PO (2016) Action recognition from depth maps using deep convolutional neural networks. IEEE Trans Hum Mach Syst 46(4):498–509

    Article  Google Scholar 

  48. Wu W, Li C, Cheng Z, Zhang X, Jin L (2017) Yolse: egocentric fingertip detection from single rgb images. In: 2017 IEEE international conference on computer vision workshops (ICCVW). pp 623–630

  49. Wu XY (2020) A hand gesture recognition algorithm based on dc-cnn. Multimedia Tools Appl 79(13):9193–9205

    Article  Google Scholar 

  50. Xiao X, Yang Y, Ahmad T, Jin L, Chang T (2017) Design of a very compact cnn classifier for online handwritten chinese character recognition using dropweight and global pooling. In: 2017 14th IAPR international conference on document analysis and Recognition (ICDAR), vol. 01. pp 891–895

  51. Xu S, Xue Y (2017) A long term memory recognition framework on multi-complexity motion gestures. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol. 01. pp 201–205

  52. Yana B, Onoye T (2018) Recognition based on fusion network for learning spatial and temporal features. IEICE Trans Fundam Electron Commun Comput Sci E 101(11):1737–1744

    Article  Google Scholar 

  53. Yang W, Jin L, Tao D, Xie Z, Feng Z (2016) Dropsample: a new training method to enhance deep convolutional neural networks for large-scale unconstrained handwritten chinese character recognition. Pattern Recogn 58:190–203

    Article  Google Scholar 

  54. Zhang J, Jun Du, Zhang S, Liu D, Yulong Hu, Jinshui Hu, Wei Si, Dai L (2017) Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn 71:196–206

    Article  Google Scholar 

  55. Zhang X, Ye Z, Jin L, Feng Z, Xu S (2013) A new writing experience: finger writing in the air using a kinect sensor. IEEE Multimedia 20(4):85–93

    Article  Google Scholar 

  56. Zhangjie F, Xu J, Zhu Z, Liu AX, Sun X (2018) Writing in the air with wifi signals for virtual reality devices. IEEE Trans Mobile Comput 18(2):473–484

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Information Technology, Dharwad, India

    Shobhan Kumar & Arun Chauhan

  2. National Institute of Technology Agartala, Agartala, India

    Munesh Chandra Trivedi

Authors
  1. Shobhan Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Munesh Chandra Trivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arun Chauhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shobhan Kumar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, S., Chandra Trivedi, M. & Chauhan, A. Real Time Air-Written Mathematical Expression Recognition for Children’s Enhanced Learning. Neural Process Lett 55, 3355–3375 (2023). https://doi.org/10.1007/s11063-022-11012-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11012-3

Keywords

Search

Navigation