Skip to main content
SpringerLink
Log in

Touching the audience: musical haptic wearables for augmented and participatory live music performances

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

This paper introduces the musical haptic wearables for audiences (MHWAs), a class of wearable devices for musical applications targeting audiences of live music performances. MHWAs are characterized by embedded intelligence, wireless connectivity to local and remote networks, a system to deliver haptic stimuli, and tracking of gestures and/or physiological parameters. They aim to enrich musical experiences by leveraging the sense of touch as well as providing new capabilities for creative participation. The embedded intelligence enables the communication with other external devices, processes input data, and generates music-related haptic stimuli. We validate our vision with two concert-experiments. The first experiment involved a duo of electronic music performers and twenty audience members. Half of the audience used an armband-based prototype of MHWA delivering vibro-tactile feedback in response to performers’ actions on their digital musical instruments, and the other half was used as a control group. In the second experiment, a smart mandolin performer played live for twelve audience members wearing a gilet-based MHWA, which provided vibro-tactile sensations in response to the performed music. Overall, results from both experiments suggest that MHWAs have the potential to enrich the experience of listening to live music in terms of arousal, valence, enjoyment, and engagement. Nevertheless, results showed that the audio-haptic experience was not homogeneous across participants, who could be grouped as those appreciative of the vibrations and those less appreciative of them. The causes for a lack of appreciation of the haptic experience were mainly identified as the sensation of unpleasantness caused by the vibrations in certain parts of the body and the lack of the comprehension of the relation between what was felt and what was heard. Based on the reported results, we offer suggestions for practitioners interested in designing wearables for enriching the musical experience of audiences of live music via the sense of touch. Such suggestions point towards the need of mechanisms of personalization, systems able to minimize the latency between the sound and the vibrations, and a time of adaptation to the vibrations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Armitage J, Ng K (2013) Augmented opera performance. In: Information technologies for performing arts, media access, and entertainment, pp. 276–287. Springer

  2. Armitage J, Ng K (2015) Configuring a haptic interface for music performance. In: Proceedings of the Conference on Electronic Visualisation and the Arts, pp. 41–45. British Computer Society

  3. Armitage J, Ng K (2015) Feeling sound: exploring a haptic-audio relationship. In: International Symposium on Computer Music Multidisciplinary Research, pp. 146–152. Springer, Cham. https://doi.org/10.1007/978-3-319-46282-0_9

  4. Armitage J, Ng K (2014) Tactile composition: configurations and communications for a musical haptic chair. In: Proceedings of the International Computer Music Conference

  5. Boer L, Cahill B, Vallgårda A (2017) The hedonic haptics player: a wearable device to experience vibrotactile compositions. In: Proceedings of the 2017 ACM Conference Companion Publication on Designing Interactive Systems, pp. 297–300

  6. Bouwer A, Holland S, Dalgleish M (2013) The haptic bracelets: learning multi-limb rhythm skills from haptic stimuli while reading. In: Music and human-computer interaction, pp. 101–122. Springer

  7. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101

    Article  Google Scholar 

  8. Bryan-Kinns N, Hamilton F (2012) Identifying mutual engagement. Behaviour & Information Technology 31(2):101–125. https://doi.org/10.1080/01449290903377103

    Article  Google Scholar 

  9. Capra O, Berthaut F, Grisoni L (2017) Toward augmented familiarity of the audience with digital musical instruments. In: International symposium on computer music multidisciplinary research, pp. 387–398

  10. Chang A, O’Sullivan C (2008) An audio-haptic aesthetic framework influenced by visual theory. In: International workshop on haptic and audio interaction design, pp. 70–80. Springer

  11. Diaconu M (2006) Reflections on an aesthetics of touch, smell and taste. Contemp Aesthet 4(1):8

    Google Scholar 

  12. Eid M, Al Osman H (2016) Affective haptics: current research and future directions. IEEE Access 4:26–40

    Article  Google Scholar 

  13. Fazekas G, Barthet M, Sandler M (2013) The Mood Conductor system: audience and performer interaction using mobile technology and emotion cues. In: Proceedings of the International Symposium on Computer Music Multidisciplinary Research

  14. Fels S, Gadd A, Mulder A (2002) Mapping transparency through metaphor: towards more expressive musical instruments. Organised Sound 7(2):109–126. https://doi.org/10.1017/S1355771802002042

    Article  Google Scholar 

  15. Freeman J (2008) Extreme sight-reading, mediated expression, and audience participation: real-time music notation in live performance. Comput Music J 32(3):25–41

    Article  Google Scholar 

  16. Frid E, Giordano M, Schumacher M, Wanderley M (2014) Physical and perceptual characterization of a tactile display for a live-electronics notification system. In: Proceedings of the International Computer Music and Sound and Music Computing Joint Conference. McGill University

  17. Gabrielli L, Squartini S (2016) Wireless Networked Music Performance. Springer, Singapore. https://doi.org/10.1007/978-981-10-0335-6_5

    Book  Google Scholar 

  18. Golz P, Shaw A (2014) Augmenting live performance dance through mobile technology. In: Proceedings of the International BCS Human Computer Interaction Conference on HCI, pp. 311–316

  19. Gunther E, O’Modhrain S (2003) Cutaneous grooves: composing for the sense of touch. J New Music Res 32(4):369–381. https://doi.org/10.1076/jnmr.32.4.369.18856

    Article  Google Scholar 

  20. Hayes L (2015) Skin music (2012) an audio-haptic composition for ears and body.. In: Proceedings of the Conference on Creativity and Cognition, pp. 359–360. https://doi.org/10.1145/2757226.2757370

  21. Hayes L, Rajko J (2017) Towards an aesthetics of touch. In: Proceedings of the International Conference on Movement Computing, pp. 22:1–22:8. ACM. https://doi.org/10.1145/3077981.3078028

  22. Hödl O (2016) The design of technology-mediated audience participation in live music. Ph.D. thesis, Vienna University of Technology

  23. Holland S, Bouwer A, Hödl O (2018) Haptics for the development of fundamental rhythm skills, including multi-limb coordination. In: S. Papetti, C. Saitis eds. Musical Haptics. Springer Series on Touch and Haptic Systems. Springer International Publishing

  24. Hwang I, Lee H, Choi S (2013) Real-time dual-band haptic music player for mobile devices. IEEE Trans Haptics 6(3):340–351

    Article  Google Scholar 

  25. Ignoto P, Hattwick I, Wanderley M (2017) Development of a vibrotactile metronome to assist in conducting contemporary classical music. In: International conference on applied human factors and ergonomics, pp. 248–258. Springer

  26. Johnson RM, van der Linden J, Rogers Y (2010) Musicjacket: the efficacy of real-time vibrotactile feedback for learning to play the violin. In: CHI’10 Extended abstracts on human factors in computing systems, pp. 3475–3480

  27. Karam M, Branje C, Nespoli G, Thompson N, Russo FA, Fels DI (2010) The emoti-chair: an interactive tactile music exhibit. In: CHI’10 Extended abstracts on human factors in computing systems, pp. 3069–3074. ACM

  28. Kitagawa N, Spence C (2006) Audiotactile multisensory interactions in human information processing. Jpn Psychol Res 48(3):158–173

    Article  Google Scholar 

  29. Lederman S, Klatzky R (2009) Haptic perception: a tutorial. Atten Percept Psycho 71(7):1439–1459

    Article  Google Scholar 

  30. Lederman SJ, Jones LA (2011) Tactile and haptic illusions. IEEE Transactions on Haptics 4(4):273–294

    Article  Google Scholar 

  31. Marshall M, Wanderley M (2011) Examining the effects of embedded vibrotactile feedback on the feel of a digital musical instrument. In: Proceedings of the Conference on New Interfaces for Musical Expression, pp. 399–404

  32. Mazzanti D, Zappi V, Caldwell D, Brogni A (2014) Augmented stage for participatory performances. In: Proceedings of the Conference on New Interfaces for Musical Expression, pp. 29–34

  33. Mazzoni A, Bryan-Kinns N (2016) Mood glove: a haptic wearable prototype system to enhance mood music in film. Entertainment Computing 17:9–17. https://doi.org/10.1016/j.entcom.2016.06.002

    Article  Google Scholar 

  34. McDowell JA, Furlong DJ (2018) Haptic-listening and the classical guitar. In: Proceedings of the Conference on New Interfaces for Musical Expression

  35. McPherson A, Zappi V (2015) An environment for submillisecond-latency audio and sensor processing on BeagleBone black.. In: Audio Engineering Society Convention 138. Audio Engineering Society. http://www.aes.org/e-lib/browse.cfm?elib=17755

  36. Merchel S, Altinsoy M (2009) Vibratory and acoustical factors in multimodal reproduction of concert dvds. In: International conference on haptic and audio interaction design, pp. 119–127. Springer

  37. Merchel S, Altinsoy ME (2018) Auditory-tactile experience of music. . In: S. Papetti, C. Saitis eds. Musical Haptics, pp. 123–148. Springer. https://doi.org/10.1007/978-3-319-58316-7_7

  38. Michailidis T (2016) On the hunt for feedback: vibrotactile feedback in interactive electronic music performances, Ph.D. thesis, Birmingham City University

  39. Miranda E, Wanderley M (2006) New digital musical instruments: control and interaction beyond the keyboard, vol. 21. AR Editions, Inc

  40. Nanayakkara S, Taylor E, Wyse L, Ong S (2009) An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 337–346. ACM

  41. Ng K (2001) Augmented stages for installation-arts and performance Proceedings of Music without Walls

  42. Papetti S, Saitis C (2018) (eds.): Musical haptics. Springer Series on Touch and Haptic Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-58316-7

  43. Paterson M (2007) The senses of touch: haptics, affects and technologies Berg

  44. Puckette M, Apel T, Ziccarelli D (1998) Real-time audio analysis tools for pd and msp. In: Proceedings of the International Computer Music Conference

  45. Rottondi C, Chafe C, Allocchio C, Sarti A (2016) An overview on networked music performance technologies. IEEE Access 4:8823–8843. https://doi.org/10.1109/ACCESS.2016.2628440

    Article  Google Scholar 

  46. Schloss WA (2003) Using contemporary technology in live performance: the dilemma of the performer. Journal of New Music Research 32(3):239–242. https://doi.org/10.1076/jnmr.32.3.239.16866

    Article  Google Scholar 

  47. Sheridan J, Bryan-Kinns N (2008) Designing for performative tangible interaction. International Journal of Arts and Technology 1(3-4):288–308

    Article  Google Scholar 

  48. Sparacino F, Wren C, Davenport G, Pentland A (1999) Augmented performance in dance and theater. International Dance and Technology 99:25–28

    Google Scholar 

  49. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psycho 73(4):971–995

    Article  Google Scholar 

  50. Turchet L (2018) Smart mandolin: autobiographical design, implementation, use cases, and lessons learned.. In: Proceedings of Audio Mostly Conference, pp. 13:1–13:7. https://doi.org/10.1145/3243274.3243280

  51. Turchet L (2018) Some reflections on the relation between augmented and smart musical instruments.. In: Proceedings of Audio Mostly Conference, pp. 17:1–17:7. https://doi.org/10.1145/3243274.3243281

  52. Turchet L (2019) Smart musical instruments: vision, design principles, and future directions. IEEE Access 7:8944–8963. https://doi.org/10.1109/ACCESS.2018.2876891

    Article  Google Scholar 

  53. Turchet L, Barthet M (2019) Co-design of musical haptic wearables for electronic music performer’s communication. IEEE T Hum-Mach Syst 49(2):183–193. https://doi.org/10.1109/THMS.2018.2885408

    Article  Google Scholar 

  54. Turchet L, Barthet M (2019) Haptification of performer’s control gestures in live electronic music performance. In: Proceedings of audio mostly conference, pp 244–247

  55. Turchet L, Fischione C, Essl G, Keller D, Barthet M (2018) Internet of Musical Things: vision and challenges. IEEE Access 6:61994–62017. https://doi.org/10.1109/ACCESS.2018.2872625

    Article  Google Scholar 

  56. Turchet L, West T, Wanderley MM (2019) Smart mandolin and musical haptic gilet: effects of vibro-tactile stimuli during live music performance. In: Proceedings of audio mostly conference, pp 168–175

  57. Turino T (2008) Music as social life: the politics of participation university of chicago press

  58. West T, Bachmayer A, Bhagwati S, Berzowska J, Wanderley MM (2019) The design of the body::suit::score, a full-body vibrotactile musical score.. In: S. Yamamoto, H. Mori (eds.) Human interface and the management of information. Information in Intelligent Systems, HCII 2019. Lecture Notes in Computer Science, vol. 11570, pp. 70–89. Springer-Verlag, Cham. https://doi.org/10.1007/978-3-030-22649-7_7

  59. Wu Y, Zhang L, Bryan-Kinns N, Barthet M (2017) Open symphony: creative participation for audiences of live music performances. IEEE MultiMedia 24(1):48–62

    Article  Google Scholar 

Download references

Acknowledgments

The authors are grateful to Dr. Mathieu Barthet for the fruitful discussions concerning some ideas of this paper.

Funding

Luca Turchet acknowledges support from the EU MSCA Individual fellowship (749561), Marcelo M. Wanderley from the Natural Sciences and Engineering Research of Canada Discovery grant. The authors acknowledge support from the SSHRC Insight grant (principal investigator is Sandeep Bhagwati) that funded the development of the musical haptic gilets.

Author information

Authors and Affiliations

  1. University of Trento, Trento, Italy

    Luca Turchet

  2. McGill University, Montreal, Quebec, Canada

    Travis West & Marcelo M. Wanderley

Authors
  1. Luca Turchet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Travis West
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcelo M. Wanderley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luca Turchet.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Turchet, L., West, T. & Wanderley, M.M. Touching the audience: musical haptic wearables for augmented and participatory live music performances. Pers Ubiquit Comput 25, 749–769 (2021). https://doi.org/10.1007/s00779-020-01395-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-020-01395-2

Keywords

Search

Navigation