Abstract
Learning music is a challenging process that requires years of practice to master, either with lessons from a professional teacher or through self-teaching. While practicing, students are expected to self-evaluate their performance which may be difficult without timely feedback from a professional. Research into computer-assisted music instrument tutoring (CAMIT) attempts to address this through the use of emerging technologies. In this paper, we study CAMIT for mixed reality (MR) by developing MR:emin, an immersive MR music learning environment for the theremin, an electronic music instrument that is controlled without physical contact. MR:emin integrates a physical theremin with the immersive learning environment. To better understand the effectiveness of such environments, we perform a user study with MR:emin comparing traditional music learning with two virtual learning environments, an immersive one and a non-immersive one. In a between-groups study, 30 participants were trained to play a sequence of notes on the theremin using one of the three training environments. Results of our statistical analysis show that performance error during training is significantly smaller in the immersive MR environment. This does not necessarily lead to improved performance after training; analysis of post-training improvement indicates that immersive training results in the smallest amount of improvement. Participants, however, indicate that the MR:emin environment is more engaging and increases confidence during practice. We discuss potential factors leading to the decrease in learning and provide some environment guidelines to aid in the design of engaging immersive music learning environments.
Similar content being viewed by others
References
Adamovich SV, Fluet GG, Tunik E, Merians AS (2009) Sensorimotor training in virtual reality: a review. NeuroRehabilitation 25(1):29–44
Berthaut DF, Desainte-Catherine M, Hachet M (2011) Interacting with 3d reactive widgets for musical performance. J New Music Res 40(3):253–263
Borsci S, Lawson G, Jha B, Burges M, Salanitri D (2016) Effectiveness of a multidevice 3D virtual environment application to train car service maintenance procedures. Virtual Real 20(1):41–55
Carlson P, Peters A, Gilbert SB, Vance JM, Luse A (2015) Virtual training: learning transfer of assembly tasks. IEEE Trans Vis Comput Graph 21(6):770–782
Chow J, Feng H, Amor R, Wünsche BC (2013) Music education using augmented reality with a head mounted display. In: Proceedings of the fourteenth Australasian user interface conference, Australian Computer Society, Inc., Darlinghurst, Australia, AUIC ’13, pp 73–79
Cormier SM, Hagman JD (1987) Transfer of learning: contemporary research and applications. Academic Press, San Diego
Dalgarno B, Lee MJ (2010) What are the learning affordances of 3-D virtual environments? Br J Educ Technol 41(1):10–32
Dalmazzo DC, Ramirez R (2019) Bowing gestures classification in violin performance: a machine learning approach. Front Psychol 10:344
Dannenberg RB, Sanchez M, Joseph A, Capell P, Joseph R, Saul R (1990) A computer based multimedia tutor for beginning piano students. Interface 19(2–3):155–173
Dannenberg RB, Sanchez M, Joseph A, Joseph R, Saul R, Capell P (1993) Results from the piano tutor project. In: Proceedings of the fourth biennial arts and technology symposium, pp 143–150
Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6(3):241–252
Gelineck S, Böttcher N, Martinussen L, Serafin S (2005) Virtual reality instruments capable of changing dimensions in real-time. Enactive05
Giraldo S, Waddell G, Nou I, Ortega A, Mayor O, Perez A, Williamon A, Ramirez R (2019) Automatic assessment of tone quality in violin music performance. Front Psychol 10:334
Gonzalez-Franco M, Pizarro R, Cermeron J, Li K, Thorn J, Hutabarat W, Tiwari A, Bermell-Garcia P (2017) Immersive mixed reality for manufacturing training. Front Robot AI 4:3
Hart SG (2006) Nasa-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50, no 9, pp 904–908
Huang F, Zhou Y, Yu Y, Wang Z, Du S (2011) Piano ar: a markerless augmented reality based piano teaching system. In: 2011 international conference on intelligent human–machine systems and cybernetics, vol 2, pp 47–52
John NW, Phillips NI, La C, Pop SR, Coope D, Kamaly-Asl I, de Souza C, Watt SJ (2016) The use of stereoscopy in a neurosurgery training virtual environment. Presence Teleoper Virtual Environ 25(4):289–298
John NW, Pop SR, Day TW, Ritsos PD, Headleand CJ (2018) The implementation and validation of a virtual environment for training powered wheelchair manoeuvres. IEEE Trans Vis Comput Graph 24(5):1867–1878
Johnson D, Tzanetakis G (2017) Vrmin: using mixed reality to augment the theremin for musical tutoring. In: Proceedings of the 2017 conference on new interfaces for musical expression
Johnson D, Dufour I, Tzanetakis G, Damien D (2016) Detecting pianist hand posture mistakes for virtual piano tutoring. In: Proceedings of the international computer music conference
Kweon Y, Kim S, Yoon B, Jo T, Park C (2018) Implementation of educational drum contents using mixed reality and virtual reality. In: Stephanidis C (ed) HCI International 2018—Posters’ Extended Abstracts. Springer, Cham, pp 296–303
Lanier J (1992) The sound of one hand. http://www.jaronlanier.com/vr.html. Accessed 5 Dec 2016
Lehmann KS, Ritz JP, Maass H, Çakmak HK, Kuehnapfel UG, Germer CT, Bretthauer G, Buhr HJ (2005) A prospective randomized study to test the transfer of basic psychomotor skills from virtual reality to physical reality in a comparable training setting. Ann Surg 241(3):442
Liarokapis F (2005) Augmented reality scenarios for guitar learning. In: Lever LM, McDerby M (eds) EG UK Theory and practice of computer graphics. The Eurographics Association
Machover T, Chung JT (1989) Hyperinstruments: musically intelligent and interactive performance and creativity systems. In: Proceedings of the international computer music conference
Mäki-Patola T, Laitinen J, Kanerva A, Takala T (2005) Experiments with virtual reality instruments. In: Proceedings of the 2005 conference on new interfaces for musical expression. National University of Singapore, Singapore, Singapore, pp 11–16
Miles HC, Pop SR, Watt SJ, Lawrence GP, John NW (2012) A review of virtual environments for training in ball sports. Comput Graph 36(6):714–726, 2011 Joint Symposium on Computational Aesthetics (CAe), Non-Photorealistic Animation and Rendering (NPAR), and Sketch-Based Interfaces and Modeling (SBIM)
Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. In: Photonics for industrial applications. International Society for Optics and Photonics, pp 282–292
Moore AG, Howell MJ, Stiles AW, Herrera NS, McMahan RP (2015) Wedge: a musical interface for building and playing composition-appropriate immersive environments. In: 2015 IEEE symposium on 3D user interfaces
Mora J, s Lee W, Comeau G, Shirmohammadi S, Saddik AE (2006) Assisted piano pedagogy through 3d visualization of piano playing. In: 2006 IEEE international workshop on haptic audio visual environments and their applications, pp 157–160
Murcia-Lopez M, Steed A (2018) A comparison of virtual and physical training transfer of bimanual assembly tasks. IEEE Trans Vis Comput Graph 24(4):1574–1583
Ng KC, Weyde T, Koerselman T, Ong B, Neubarth K, Larkin O (2007) 3D augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support. In: Proceedings of the ninth international conference on multimodal interfaces. ACM Press, New York, p 339
Oren M, Carlson P, Gilbert S, Vance JM (2012) Puzzle assembly training: real world vs. virtual environment. In: Proceedings of IEEE virtual reality. IEEE, pp 27–30
Percival G, Wang Y, Tzanetakis G (2007) Effective use of multimedia for computer-assisted musical instrument tutoring. In: Proceedings of the international workshop on educational multimedia and multimedia education. ACM, pp 67–76
Raptis S, Chalamandaris A, Baxevanis A, Askenfelt A, Schoonderwaldt E, Hansen KF, Fober D, Letz S, Orlarey Y (2005) Imutus—an effective practicing environment for music tuition. In: Proceedings of the international computer music conference
Rose FD, Attree EA, Brooks BM, Parslow DM, Penn PR (2000) Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergon 43(4):494–511
Schoonderwaldt E, Askenfelt A, Hansen KF (2005) Design and implementation of automatic evaluation of recorder performance in IMUTUS. In: Proceedings of the international computer music conference, pp 97–103
Schrepp M, Hinderks A, Thomaschewski J (2014) Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In: International conference of design, user experience, and usability. Springer, pp 383–392
Serafin S, Erkut C, Kojs J, Nilsson NC, Nordahl R (2016) Virtual reality musical instruments: state of the art, design principles, and future directions. Comput Music J 41(2):22–40
Siegel S, Castellan NJ Jr (1988) Nonparametric statistics for the behavioral sciences, 2nd edn. Mcgraw-Hill Book Company, New York
Thomsen ASS, Bach-Holm D, Kjærbo H, Højgaard-Olsen K, Subhi Y, Saleh GM, Park YS, la Cour M, Konge L (2017) Operating room performance improves after proficiency-based virtual reality cataract surgery training. Ophthalmology 124(4):524–531
Waller D, Hunt E, Knapp D (1998) The transfer of spatial knowledge in virtual environment training. Presence Teleoper Virtual Environ 7(2):129–143
Werrlich S, Nguyen PA, Notni G (2018) Evaluating the training transfer of head-mounted display based training for assembly tasks. In: Proceedings of the 11th PErvasive Technologies related to assistive environments conference, PETRA ’18. ACM, New York, pp 297–302
Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence Teleoper Virtual Environ 7(3):225–240
Young D (2002) The hyperbow controller: real-time dynamics measurement of violin performance. In: Proceedings of the 2002 conference on new interfaces for musical expression
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Johnson, D., Damian, D. & Tzanetakis, G. Evaluating the effectiveness of mixed reality music instrument learning with the theremin. Virtual Reality 24, 303–317 (2020). https://doi.org/10.1007/s10055-019-00388-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-019-00388-8