Skip to main content
SpringerLink
Log in

An electromyogram-based tapping gesture model with differentiated vibration feedback by low-fidelity actuators

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Among other aspects like attention and focus, the effectiveness of actions, such as training in percussion, also relies on the correct gesture and strength of the individual carrying out the activity repetitively to develop proper muscle memory and train motor skills. To make this process economical and efficient, we propose a system that models the feature of direction and strength of a tap and receives the corresponding active haptic feedback in an immersive virtual environment. We propose a novel tapping gesture model that takes electromyogram and vibration feedback into consideration. We also propose an approach of designing vibration modes that follow the mechanical stimulation principles and generate distinguishable vibration feedback with low-fidelity actuators. We developed a prototype using myoelectric and haptic apparatus (Myo armband and HTC VIVE controllers) and evaluated our system by conducting a user study with 50 participants. The evaluation shows that the users can distinguish three common materials (wood, rubber, and aluminum), and our system allows them to actively adjust their direction and strength to grasp the correct point while tapping. We demonstrate the efficacy of our tapping model on a virtual Chinese chimes application and observe positive feedback from both novices and expert users.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.vive.com/

  2. https://developerblog.myo.com/author/thalmic-labs/

  3. https://www.ll3d.com/

References

  • Akhtar A, Hargrove LJ, Bretl T (2012) Prediction of distal arm joint angles from EMG and shoulder orientation for prosthesis control. In: 2012 annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 4160–4163

  • Asamura N, Yokoyama N, Shinoda H (1999) A method of selective stimulation to epidermal skin receptors for realistic touch feedback. In: IEEE virtual reality, pp 274–281

  • Avanzini F, Crosato P (2006) Integrating physically based sound models in a multimodal rendering architecture. Comput Animat Virtual Worlds 17(3–4):411–419

    Article  Google Scholar 

  • Camacho Navarro J, Leon-Vargas F, Barrero Pérez J (2012) EMG-based system for basic hand movement recognition. Dyna 79(171):41–49

    Google Scholar 

  • Chang GC, Kang WJ, Luh JJ, Cheng CK, Lai JS, Chen JJJ, Kuo TS (1996) Real-time implementation of electromyogram pattern recognition as a control command of man-machine interface. Med Eng Phys 18(7):529–537

    Article  Google Scholar 

  • Choi S, Kuchenbecker KJ (2013) Vibrotactile display: perception, technology, and applications. Proc IEEE 101(9):2093–2104

    Article  Google Scholar 

  • Choi K, Hirose H, Iijima T, Koike Y (2006) Prediction of four degrees of freedom arm movement using EMG signal. In: 2005 IEEE engineering in medicine and biology 27th annual conference. IEEE, pp 5820–5823

  • Cosman PH, Cregan PC, Martin CJ, Cartmill JA (2002) Virtual reality simulators: current status in acquisition and assessment of surgical skills. ANZ J Surg 72(1):30–34

    Article  Google Scholar 

  • Gescheider GA (1976) Evidence in support of the duplex theory of mechanoreception. Sensory Processes

  • Gescheider GA, Berryhill ME, Verrillo RT, Bolanowski SJ (1999) Vibrotactile temporal summation: probability summation or neural integration? Somatosens Motor Res 16(3):229–242

    Article  Google Scholar 

  • Gescheider GA, Bolanowski SJ, Pope JV, Verrillo RT (2002) A four-channel analysis of the tactile sensitivity of the fingertip: frequency selectivity, spatial summation, and temporal summation. Somatosens Motor Res 19(2):114–124

    Article  Google Scholar 

  • Giordano B, Avanzini F, Wanderley M, Mcadams S (2010) Multisensory integration in percussion performance. In: 10ème Congrès Français d’Acoustique

  • Hachisu T, Sato M, Fukushima S, Kajimoto H (2012) Augmentation of material property by modulating vibration resulting from tapping. In: International conference on human haptic sensing and touch enabled computer applications, pp 173–180

  • Hudgins B, Parker P, Scott RN (1993) A new strategy for multifunction myoelectric control. IEEE Trans Biomed Eng 40(1):82–94

    Article  Google Scholar 

  • Jones LA, Lederman SJ (2006) Human hand function. Oxford University Press, Oxford

    Book  Google Scholar 

  • Kaliki RR, Davoodi R, Loeb GE (2008) Prediction of distal arm posture in 3-d space from shoulder movements for control of upper limb prostheses. Proc IEEE 96(7):1217–1225

    Article  Google Scholar 

  • Kim D, Hilliges O, Izadi S, Butler AD, Chen J, Oikonomidis I, Olivier P (2012) Digits: freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In: Proceedings of the 25th annual ACM symposium on User interface software and technology, pp 167–176

  • Kim K, Kang J, Lee Y, Moon C, Choi H, Mun C (2011) The development of muscle training system using the electromyogram and interactive game for physical rehabilitation. In: 5th Kuala Lumpur international conference on biomedical engineering 2011. Springer, pp 801–804

  • Konyo M, Tadokoro S, Yoshida A, Saiwaki N (2005) A tactile synthesis method using multiple frequency vibrations for representing virtual touch. In: IEEE international conference on intelligent robots and systems, pp 3965–3971

  • Kuchenbecker KJ, Fiene J, Niemeyer G (2006) Improving contact realism through event-based haptic feedback. IEEE Trans Vis Comput Gr 12(2):219–230

    Article  Google Scholar 

  • Kwon C (2019) Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies. Virtual Real 23(1):101–118

    Article  Google Scholar 

  • Lamata P, Gómez E, Sánchez-Margallo F, Lamata F, Del Pozo F, Usón J (2006) Tissue consistency perception in laparoscopy to define the level of fidelity in virtual reality simulation. Surg Endosc Interv Tech 20(9):1368–1375

    Article  Google Scholar 

  • Lenzi T, De Rossi SM, Vitiello N, Carrozza MC (2012) Intention-based EMG control for powered exoskeletons. IEEE Trans Biomed Eng 59(8):2180–2190

    Article  Google Scholar 

  • Lyons G, Sharma P, Baker M, O’Malley S, Shanahan A (2003) A computer game-based EMG biofeedback system for muscle rehabilitation. In: IEEE engineering in medicine and biology society, pp 1625–1628

  • Nymoen K, Haugen MR, Jensenius AR (2015) Mumyo—evaluating and exploring the myo armband for musical interaction. In: International conference on new interfaces for musical expression, Louisiana State University

  • Okamura AM, Cutkosky MR, Dennerlein JT (2001) Reality-based models for vibration feedback in virtual environments. IEEE/ASME Trans Mechatron 6(3):245–252

    Article  Google Scholar 

  • Pacchierotti C (2015) Cutaneous haptic feedback in robotic teleoperation. Springer, Berlin

    Book  Google Scholar 

  • Pai YS, Dingler T, Kunze K (2019) Assessing hands-free interactions for VR using eye gaze and electromyography. Virtual Reality 23(2):119–131

    Article  Google Scholar 

  • Papetti S, Fröhlich M, Fontana F, Schiesser S, Avanzini F (2018) Implementation and characterization of vibrotactile interfaces. In: Musical haptics. Springer, Berlin, pp 257–282

  • Phinyomark A, Hirunviriya S, Limsakul C, Phukpattaranont P (2010) Evaluation of EMG feature extraction for hand movement recognition based on Euclidean distance and standard deviation. ECTI-CON2010: The 2010 ECTI international confernce on electrical engineering/electronics. Computer, Telecommunications and Information Technology. IEEE, pp 856–860

  • Pons JL, Ceres R, Rocon E, Levin S, Markovitz I, Saro B, Reynaerts D, Moorleghem WV, Bueno L (2005) Virtual reality training and EMG control of the MANUS hand prosthesis. Robotica 23(3):311–317

    Article  Google Scholar 

  • Pulliam CL, Lambrecht JM, Kirsch RF (2011) Electromyogram-based neural network control of transhumeral prostheses. J Rehabil Res Dev 48(6):739

    Article  Google Scholar 

  • Reaz MBI, Hussain MS, Mohd-Yasin F (2006) Techniques of EMG signal analysis: detection, processing, classification and applications. Biol Proced Online 8(1):11–35

    Article  Google Scholar 

  • Ryge A, Thomsen L, Berthelsen T, Hvass JS, Koreska L, Vollmers C, Nilsson NC, Nordahl R, Serafin S (2017) Effect on high versus low fidelity haptic feedback in a virtual reality baseball simulation. In: IEEE virtual reality, pp 365–366

  • Saponas TS, Tan DS, Morris D, Balakrishnan R, Turner J, Landay JA (2009) Enabling always-available input with muscle-computer interfaces. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology, pp 167–176

  • Saponas TS, Tan DS, Morris D, Turner J, Landay JA (2010) Making muscle-computer interfaces more practical. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 851–854

  • Shima K, Tsuji T (2009) An MMG-based human-assisting manipulator using acceleration sensors. In: 2009 IEEE international conference on systems, man and cybernetics. IEEE, pp 2433–2438

  • Ström P, Hedman L, Särnå L, Kjellin A, Wredmark T, Felländer-Tsai L (2006) Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc Interv Tech 20(9):1383–1388

    Article  Google Scholar 

  • Van der Meijden OA, Schijven MP (2009) The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg Endosc 23(6):1180–1190

    Article  Google Scholar 

  • Wellman P, Howe RD (1995) Towards realistic vibrotactile display in virtual environments. ASME Dyn Syst Control Div 57:713–718

    Google Scholar 

  • Westling G, Johansson RS (1987) Responses in glabrous skin mechanoreceptors during precision grip in humans. Exp Brain Res 66(1):128–140

    Article  Google Scholar 

  • Wirth M, Grad S, Poimann D, Richer R, Ottmann J, Eskofier BM (2018) Exploring the feasibility of EMG based interaction for assessing cognitive capacity in virtual reality. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4953–4956

  • Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240

    Article  Google Scholar 

  • Wu CM, Hsu CW, Lee TK, Smith S (2017) A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment. Virtual Real 21(1):19–29

    Article  Google Scholar 

  • Yem V, Kajimoto H (2017) Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world. In: IEEE virtual reality, pp 99–104

Download references

Acknowledgements

This work was supported by National Key R&D Program of China (No.2017YFB1402105, 2017YFB1002604), National Key Cooperation between the BRICS Program of China(No.2017YFE0100500) and Beijing Natural Science Foundation of China (No. 4172033).

Author information

Author notes

  1. Xingce Wang and Yan Fu have contributed equally to this work.

Authors and Affiliations

  1. School of Artificial Intelligence, Beijing Normal University, Beijing, China

    Pengfei Xu, Zhongke Wu, Xingce Wang & Yan Fu

  2. School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore

    Ying He

Authors
  1. Pengfei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongke Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingce Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhongke Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 284061 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, P., Wu, Z., Wang, X. et al. An electromyogram-based tapping gesture model with differentiated vibration feedback by low-fidelity actuators. Virtual Reality 25, 383–397 (2021). https://doi.org/10.1007/s10055-020-00458-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-020-00458-2

Keywords

Search

Navigation