Skip to main content
Log in

Evaluating the effectiveness of mixed reality music instrument learning with the theremin

  • S.I. : XR (VR, AR, MR) and Immersive Learning Environments
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Learning music is a challenging process that requires years of practice to master, either with lessons from a professional teacher or through self-teaching. While practicing, students are expected to self-evaluate their performance which may be difficult without timely feedback from a professional. Research into computer-assisted music instrument tutoring (CAMIT) attempts to address this through the use of emerging technologies. In this paper, we study CAMIT for mixed reality (MR) by developing MR:emin, an immersive MR music learning environment for the theremin, an electronic music instrument that is controlled without physical contact. MR:emin integrates a physical theremin with the immersive learning environment. To better understand the effectiveness of such environments, we perform a user study with MR:emin comparing traditional music learning with two virtual learning environments, an immersive one and a non-immersive one. In a between-groups study, 30 participants were trained to play a sequence of notes on the theremin using one of the three training environments. Results of our statistical analysis show that performance error during training is significantly smaller in the immersive MR environment. This does not necessarily lead to improved performance after training; analysis of post-training improvement indicates that immersive training results in the smallest amount of improvement. Participants, however, indicate that the MR:emin environment is more engaging and increases confidence during practice. We discuss potential factors leading to the decrease in learning and provide some environment guidelines to aid in the design of engaging immersive music learning environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://company.yousician.com/.

  2. https://www.skoove.com/.

  3. https://www.moogmusic.com/products/etherwave-theremins/theremini.

  4. https://github.com/Microsoft/MixedRealityToolkit-Unity.

References

  • Adamovich SV, Fluet GG, Tunik E, Merians AS (2009) Sensorimotor training in virtual reality: a review. NeuroRehabilitation 25(1):29–44

    Article  Google Scholar 

  • Berthaut DF, Desainte-Catherine M, Hachet M (2011) Interacting with 3d reactive widgets for musical performance. J New Music Res 40(3):253–263

    Article  Google Scholar 

  • Borsci S, Lawson G, Jha B, Burges M, Salanitri D (2016) Effectiveness of a multidevice 3D virtual environment application to train car service maintenance procedures. Virtual Real 20(1):41–55

    Article  Google Scholar 

  • Carlson P, Peters A, Gilbert SB, Vance JM, Luse A (2015) Virtual training: learning transfer of assembly tasks. IEEE Trans Vis Comput Graph 21(6):770–782

    Article  Google Scholar 

  • Chow J, Feng H, Amor R, Wünsche BC (2013) Music education using augmented reality with a head mounted display. In: Proceedings of the fourteenth Australasian user interface conference, Australian Computer Society, Inc., Darlinghurst, Australia, AUIC ’13, pp 73–79

  • Cormier SM, Hagman JD (1987) Transfer of learning: contemporary research and applications. Academic Press, San Diego

    Google Scholar 

  • Dalgarno B, Lee MJ (2010) What are the learning affordances of 3-D virtual environments? Br J Educ Technol 41(1):10–32

    Article  Google Scholar 

  • Dalmazzo DC, Ramirez R (2019) Bowing gestures classification in violin performance: a machine learning approach. Front Psychol 10:344

    Article  Google Scholar 

  • Dannenberg RB, Sanchez M, Joseph A, Capell P, Joseph R, Saul R (1990) A computer based multimedia tutor for beginning piano students. Interface 19(2–3):155–173

    Article  Google Scholar 

  • Dannenberg RB, Sanchez M, Joseph A, Joseph R, Saul R, Capell P (1993) Results from the piano tutor project. In: Proceedings of the fourth biennial arts and technology symposium, pp 143–150

  • Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6(3):241–252

    Article  Google Scholar 

  • Gelineck S, Böttcher N, Martinussen L, Serafin S (2005) Virtual reality instruments capable of changing dimensions in real-time. Enactive05

  • Giraldo S, Waddell G, Nou I, Ortega A, Mayor O, Perez A, Williamon A, Ramirez R (2019) Automatic assessment of tone quality in violin music performance. Front Psychol 10:334

    Article  Google Scholar 

  • Gonzalez-Franco M, Pizarro R, Cermeron J, Li K, Thorn J, Hutabarat W, Tiwari A, Bermell-Garcia P (2017) Immersive mixed reality for manufacturing training. Front Robot AI 4:3

    Article  Google Scholar 

  • Hart SG (2006) Nasa-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50, no 9, pp 904–908

  • Huang F, Zhou Y, Yu Y, Wang Z, Du S (2011) Piano ar: a markerless augmented reality based piano teaching system. In: 2011 international conference on intelligent human–machine systems and cybernetics, vol 2, pp 47–52

  • John NW, Phillips NI, La C, Pop SR, Coope D, Kamaly-Asl I, de Souza C, Watt SJ (2016) The use of stereoscopy in a neurosurgery training virtual environment. Presence Teleoper Virtual Environ 25(4):289–298

    Article  Google Scholar 

  • John NW, Pop SR, Day TW, Ritsos PD, Headleand CJ (2018) The implementation and validation of a virtual environment for training powered wheelchair manoeuvres. IEEE Trans Vis Comput Graph 24(5):1867–1878

    Article  Google Scholar 

  • Johnson D, Tzanetakis G (2017) Vrmin: using mixed reality to augment the theremin for musical tutoring. In: Proceedings of the 2017 conference on new interfaces for musical expression

  • Johnson D, Dufour I, Tzanetakis G, Damien D (2016) Detecting pianist hand posture mistakes for virtual piano tutoring. In: Proceedings of the international computer music conference

  • Kweon Y, Kim S, Yoon B, Jo T, Park C (2018) Implementation of educational drum contents using mixed reality and virtual reality. In: Stephanidis C (ed) HCI International 2018—Posters’ Extended Abstracts. Springer, Cham, pp 296–303

    Chapter  Google Scholar 

  • Lanier J (1992) The sound of one hand. http://www.jaronlanier.com/vr.html. Accessed 5 Dec 2016

  • Lehmann KS, Ritz JP, Maass H, Çakmak HK, Kuehnapfel UG, Germer CT, Bretthauer G, Buhr HJ (2005) A prospective randomized study to test the transfer of basic psychomotor skills from virtual reality to physical reality in a comparable training setting. Ann Surg 241(3):442

    Article  Google Scholar 

  • Liarokapis F (2005) Augmented reality scenarios for guitar learning. In: Lever LM, McDerby M (eds) EG UK Theory and practice of computer graphics. The Eurographics Association

  • Machover T, Chung JT (1989) Hyperinstruments: musically intelligent and interactive performance and creativity systems. In: Proceedings of the international computer music conference

  • Mäki-Patola T, Laitinen J, Kanerva A, Takala T (2005) Experiments with virtual reality instruments. In: Proceedings of the 2005 conference on new interfaces for musical expression. National University of Singapore, Singapore, Singapore, pp 11–16

  • Miles HC, Pop SR, Watt SJ, Lawrence GP, John NW (2012) A review of virtual environments for training in ball sports. Comput Graph 36(6):714–726, 2011 Joint Symposium on Computational Aesthetics (CAe), Non-Photorealistic Animation and Rendering (NPAR), and Sketch-Based Interfaces and Modeling (SBIM)

  • Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. In: Photonics for industrial applications. International Society for Optics and Photonics, pp 282–292

  • Moore AG, Howell MJ, Stiles AW, Herrera NS, McMahan RP (2015) Wedge: a musical interface for building and playing composition-appropriate immersive environments. In: 2015 IEEE symposium on 3D user interfaces

  • Mora J, s Lee W, Comeau G, Shirmohammadi S, Saddik AE (2006) Assisted piano pedagogy through 3d visualization of piano playing. In: 2006 IEEE international workshop on haptic audio visual environments and their applications, pp 157–160

  • Murcia-Lopez M, Steed A (2018) A comparison of virtual and physical training transfer of bimanual assembly tasks. IEEE Trans Vis Comput Graph 24(4):1574–1583

    Article  Google Scholar 

  • Ng KC, Weyde T, Koerselman T, Ong B, Neubarth K, Larkin O (2007) 3D augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support. In: Proceedings of the ninth international conference on multimodal interfaces. ACM Press, New York, p 339

  • Oren M, Carlson P, Gilbert S, Vance JM (2012) Puzzle assembly training: real world vs. virtual environment. In: Proceedings of IEEE virtual reality. IEEE, pp 27–30

  • Percival G, Wang Y, Tzanetakis G (2007) Effective use of multimedia for computer-assisted musical instrument tutoring. In: Proceedings of the international workshop on educational multimedia and multimedia education. ACM, pp 67–76

  • Raptis S, Chalamandaris A, Baxevanis A, Askenfelt A, Schoonderwaldt E, Hansen KF, Fober D, Letz S, Orlarey Y (2005) Imutus—an effective practicing environment for music tuition. In: Proceedings of the international computer music conference

  • Rose FD, Attree EA, Brooks BM, Parslow DM, Penn PR (2000) Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergon 43(4):494–511

    Article  Google Scholar 

  • Schoonderwaldt E, Askenfelt A, Hansen KF (2005) Design and implementation of automatic evaluation of recorder performance in IMUTUS. In: Proceedings of the international computer music conference, pp 97–103

  • Schrepp M, Hinderks A, Thomaschewski J (2014) Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In: International conference of design, user experience, and usability. Springer, pp 383–392

  • Serafin S, Erkut C, Kojs J, Nilsson NC, Nordahl R (2016) Virtual reality musical instruments: state of the art, design principles, and future directions. Comput Music J 41(2):22–40

    Article  Google Scholar 

  • Siegel S, Castellan NJ Jr (1988) Nonparametric statistics for the behavioral sciences, 2nd edn. Mcgraw-Hill Book Company, New York

    Google Scholar 

  • Thomsen ASS, Bach-Holm D, Kjærbo H, Højgaard-Olsen K, Subhi Y, Saleh GM, Park YS, la Cour M, Konge L (2017) Operating room performance improves after proficiency-based virtual reality cataract surgery training. Ophthalmology 124(4):524–531

    Article  Google Scholar 

  • Waller D, Hunt E, Knapp D (1998) The transfer of spatial knowledge in virtual environment training. Presence Teleoper Virtual Environ 7(2):129–143

    Article  Google Scholar 

  • Werrlich S, Nguyen PA, Notni G (2018) Evaluating the training transfer of head-mounted display based training for assembly tasks. In: Proceedings of the 11th PErvasive Technologies related to assistive environments conference, PETRA ’18. ACM, New York, pp 297–302

  • Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence Teleoper Virtual Environ 7(3):225–240

    Article  Google Scholar 

  • Young D (2002) The hyperbow controller: real-time dynamics measurement of violin performance. In: Proceedings of the 2002 conference on new interfaces for musical expression

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to David Johnson.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Johnson, D., Damian, D. & Tzanetakis, G. Evaluating the effectiveness of mixed reality music instrument learning with the theremin. Virtual Reality 24, 303–317 (2020). https://doi.org/10.1007/s10055-019-00388-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-019-00388-8

Keywords

Navigation