Skip to main content
SpringerLink
Log in

Exploring visual stimuli as a support for novices’ creative engagement with digital musical interfaces

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Visual materials are a widely used tool for stimulating creativity. This paper explores the potential for visual stimuli to support novices’ creative engagement with multimodal digital musical interfaces. An empirical study of 24 participants was conducted to compare the effect of abstract and literal forms of graphical scores on novices’ creative engagement, and whether being informed or uninformed about meanings of symbols in the score had any impact on creative engagement. The results suggest that abstract visual stimuli can provide an effective scaffold for creative engagement when participants are not informed about their design. It was found that providing information about visual stimuli has both advantages and disadvantages, depending largely on the stimuli’s visual style. Being informed about the meaning of a literal visual stimuli helped participants in making interpretations and gaining inspiration, whereas having information about abstract stimuli led to frustration. Qualitative data indicates that both forms of visual stimuli support creative engagement but at different stages of a creative process, and a descriptive model is presented to explain this. The findings highlight the benefits of visual stimuli in supporting creative engagement in the process of music making – a multimodal interaction domain typically involving few or no visual activities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Benedetti L, Winnemöller H, Corsini M, Scopigno R (2014) Painting with bob: Assisted creativity for novices. In: Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 419–428. ACM

  2. Besada JL, Cánovas CP (2020) Timelines in spectral composition: a cognitive approach to musical creativity. Organ Sound 25(2):142–155

    Article  Google Scholar 

  3. Bilda Z, Edmonds E, Candy L (2008) Designing for creative engagement. Des Stud 29(6):525–540

    Article  Google Scholar 

  4. Borgianni Y, Maccioni L, Fiorineschi L, Rotini F (2020) Forms of stimuli and their effects on idea generation in terms of creativity metrics and non-obviousness. Int J Des Creat Innov 8(3):147–164

    Google Scholar 

  5. Bown O, Eldridge A, McCormack J (2009) Understanding interaction in contemporary digital music: from instruments to behavioural objects. Organ Sound 14(2):188–196

    Article  Google Scholar 

  6. Bryan-Kinns N (2013) Mutual engagement and collocation with shared representations. Int J Hum Comput Stud 71(1):76–90

    Article  Google Scholar 

  7. Candy L, Bilda Z (2009) Understanding and evaluating creativity. In: Proceedings of the Seventh ACM Conference on Creativity and Cognition, C &C’09, pp. 497–498. ACM, New York, NY, USA

  8. Capra O, Berthaut F, Grisoni L (2020) Levels of detail in visual augmentation for novice and expert audiences. Comput Music J 44(2–3):92–107

    Article  Google Scholar 

  9. Cardoso C, Badke-Schaub P (2011) The influence of different pictorial representations during idea generation. J Creat Behav 45(2):130–146

    Article  Google Scholar 

  10. Carroll EA (2013) Quantifying the personal creative experience: evaluation of digital creativity support tools using self-report and physiological responses. Ph.D. thesis, The University of North Carolina at Charlotte

  11. Cheng P, Mugge R, Schoormans JP (2014) A new strategy to reduce design fixation: Presenting partial photographs to designers. Des Stud 35(4):374–391

    Article  Google Scholar 

  12. Correia NN, Tanaka A (2017) Avui: Designing a toolkit for audiovisual interfaces. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1093–1104

  13. Csikszentmihalyi M (1996) Flow and the psychology of discovery and invention. Harper Collins, New York

    Google Scholar 

  14. Csikszentmihalyi M, Sawyer K (2014) Creative insight: The social dimension of a solitary moment. In: The systems model of creativity, pp. 73–98. Springer

  15. Davis N, Zook A, O’Neill B, Headrick B, Riedl M, Grosz A, Nitsche M (2013) Creativity support for novice digital filmmaking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 651–660. ACM

  16. Do Nascimento Correia N, Tanaka A (2021) From gui to avui: situating audiovisual user interfaces within human-computer interaction and related fields. EAI Endorsed Transactions on Creative Technologies 8(27)

  17. Edmonds E (2011) Art, interaction and engagement. In: Information Visualisation (IV), 2011 15th International Conference on, pp. 451–456. IEEE

  18. Edmonds E, Muller L, Connell M (2006) On creative engagement. Vis commun 5(3):307–322

    Article  Google Scholar 

  19. Fischer G (2002) Beyond ”couch potatoes”: From consumers to designers and active. In: Contributors, in FirstMonday (Peer-Reviewed Journal on the Internet), Available at \url{http://firstmonday.org/issues/issue7_12/fischer}

  20. Franceschini A, Laney R, Dobbyn C (2020) Sketching music: exploring melodic similarity and contrast using a digital tabletop. J Music Technol Educ 13(1):5–31

    Article  Google Scholar 

  21. Fu K, Chan J, Cagan J, Kotovsky K, Schunn C, Wood K (2013) The meaning of ”near” and ”far”: the impact of structuring design databases and the effect of distance of analogy on design output. Journal of Mechanical Design 135(2)

  22. Gabora L, Kaufman SB (2010) Evolutionary approaches to creativity. The Cambridge handbook of creativity pp. 279–300

  23. Glǎveanu VP, Beghetto RA (2021) Creative experience: A non-standard definition of creativity. Creat Res J 33(2):75–80. https://doi.org/10.1080/10400419.2020.1827606

    Article  Google Scholar 

  24. Goldschmidt G (2011) Avoiding design fixation: transformation and abstraction in mapping from source to target. J creat behav 45(2):92–100

    Article  Google Scholar 

  25. Goucher-Lambert K, Moss J, Cagan J (2019) A neuroimaging investigation of design ideation with and without inspirational stimuli-understanding the meaning of near and far stimuli. Des Stud 60:1–38

    Article  Google Scholar 

  26. Hansen AMS, Anderson HJ, Raudaskoski P (2011) Play fluency in music improvisation games for novices. In: Proceedings of the 11th International Conference on New Interfaces for Musical Expression, pp. 220–223

  27. Ince G, Yorganci R, Ozkul A, Duman TB, Köse H (2020) An audiovisual interface-based drumming system for multimodal human–robot interaction. Journal on Multimodal User Interfaces pp. 1–16

  28. Jensenius AR, Lyons MJ (2017) A nime reader: Fifteen years of new interfaces for musical expression, vol. 3. Springer

  29. Kerne A, Webb AM, Smith SM, Linder R, Lupfer N, Qu Y, Moeller J, Damaraju S (2014) Using metrics of curation to evaluate information-based ideation. ACM Tran Comput Hum Interact (ToCHI) 21(3):14

    Google Scholar 

  30. Kiger ME, Varpio L (2020) Thematic analysis of qualitative data: Amee guide no. 131. Medical teacher 42(8), 846–854

  31. Kim J, Bagla A, Bernstein MS (2015) Designing creativity support tools for failure. In: Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, pp. 157–160. ACM

  32. Kim J, Dontcheva M, Li W, Bernstein MS, Steinsapir D (2015) Motif: Supporting novice creativity through expert patterns. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1211–1220

  33. Klapetek A, Ngo MK, Spence C (2012) Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? Atten Percept Psychophysics 74(6):1154–1167

    Article  Google Scholar 

  34. Koh EC (2020) Read the full patent or just the claims? mitigating design fixation and design distraction when reviewing patent documents. Des Stud 68:34–57

    Article  Google Scholar 

  35. Kurtzberg TR (2005) Feeling creative, being creative: An empirical study of diversity and creativity in teams. Creat Res J 17(1):51–65

    Article  Google Scholar 

  36. Lee SW, Freeman J (2013) Real-time music notation in mixed laptop -acoustic ensembles. Comput Music J 37(4):24–36

    Article  Google Scholar 

  37. Linsey JS, Tseng I, Fu K, Cagan J, Wood KL, Schunn C (2010) A study of design fixation, its mitigation and perception in engineering design faculty. Journal of Mechanical Design 132(4)

  38. Magnusson T (2014) Improvising with the threnoscope: integrating code, hardware, gui, network, and graphic scores. In: Proceedings of the Internation Conference on New Interfaces for Musical Expression. Goldsmiths University of London

  39. Makri S, Blandford A, Woods M, Sharples S, Maxwell D (2014)“making my own luck”: Serendipity strategies and how to support them in digital information environments. J Assoc Inf Sci Technol 65(11):2179–2194

  40. Masu R, Do Nascimento Correia N, Romao T, Nime scores: a systematic review of how scores have shaped performance ecologies in nime. In: Proceedings of the 2021 International Conference on New Interfaces for Musical Expression (NIME)

  41. McCay-Peet L, Toms EG (2017) Researching serendipity in digital information environments. Synthesis Lectures on Information Concepts, Retrieval, and Services 9(6):i–91

    Article  Google Scholar 

  42. Metatla O, Correia NN, Martin F, Bryan-Kinns N, Stockman T (2016) Tap the shapetones: Exploring the effects of crossmodal congruence in an audio-visual interface. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1055–1066. ACM

  43. O’Brien HL, Toms EG (2010) The development and evaluation of a survey to measure user engagement. J Am Soc Inf Sci 61(1):50–69

    Article  Google Scholar 

  44. Polo A, Sevillano X (2019) Musical vision: an interactive bio-inspired sonification tool to convert images into music. J Multimodal User Interfaces 13(3):231–243

    Article  Google Scholar 

  45. Ramachandran VS, Hubbard EM (2001) Synaesthesia-a window into perception, thought and language. J Conscious Stud 8(12):3–34

    Google Scholar 

  46. Ramchurn R, Martinez-Avila J, Martindale S, Chamberlain A, Wilson ML, Benford S (2019) Improvising a live score to an interactive brain-controlled film. In: Proceedings of the 2019 International Conference on New Interfaces for Musical Expression (NIME)

  47. Rebelo P (2015) Composing with graphics: Revealing the compositional process through performance. In: In Proceedings of On New Tools for Music Notation and Representation

  48. Remy C, MacDonald Vermeulen L, Frich J, Biskjaer MM, Dalsgaard P (2020) Evaluating creativity support tools in hci research. In: Proceedings of the 2020 ACM Designing Interactive Systems Conference, pp. 457–476

  49. Rešidbegović D et al (2020) Composition and notation of parameters in electronic music: Approximate reductionist graphical notation. INSAM J Contemp Music Art Technol 1(4):89–111

    Google Scholar 

  50. Sanders EBN, Stappers PJ (2008) Co-creation and the new landscapes of design. Co design 4(1):5–18

    Google Scholar 

  51. Sozo V, Ogliari A (2019) Stimulating design team creativity based on emotional values: A study on idea generation in the early stages of new product development processes. Int J Ind Ergon 70:38–50

    Article  Google Scholar 

  52. Valer R, Schramm R, Nedel L (2020) Musical brush: Exploring creativity through an ar-based tool for sketching music and drawings. In: Computer Graphics International Conference, pp. 119–131. Springer

  53. Wang G (2014) Principles of visual design for computer music. In: ICMC

  54. Weinberg G (2003) Interconnected musical networks: bringing expression and thoughtfulness to collaborative group playing. Ph.D. thesis, Massachusetts Institute of Technology

  55. Wessel D, Wright M (2001) Problems and prospects for intimate musical control of computers. In: Proceedings of the CHI’01 Workshop on New Interfaces for Musical Expression (NIME-01), Seattle, USA

  56. Wong SSH, Lim SWH (2017) Mental imagery boosts music compositional creativity. PLoS ONE 12(3):e0174009

    Article  Google Scholar 

  57. Wu Y, Bryan-Kinns N (2019) Musicking with an interactive musical system: The effects of task motivation and user interface mode on non-musicians’ creative engagement. Int J Hum Comput Stud 122:61–77

    Article  Google Scholar 

  58. Xambó A, Hornecker E, Marshall P, Jordà S, Dobbyn C, Laney R (2017) Exploring social interaction with a tangible music interface. Interact Comput 29(2):248–270

    Google Scholar 

Download references

Acknowledgements

Thanks to the chief editor and reviewers for their helpful suggestions, thanks to Chuanqi Zhou and Zhuowen Shi for their assistance in data collating, and thanks to all the participants and experts for their precious time.

Funding

This paper is funded by Sichuan Modern Design and Culture Research Center (MD22E016), National Natural Science Foundation of China (52175253), Humanities and Social Sciences Project of Chinese Ministry of Education (19YJA760094), Major Project of Sichuan Social Science Research Base (SC20EZD056).

Author information

Authors and Affiliations

  1. Southwest Jiaotong University, Xian Road, Chengdu, 611756, Sichuan, China

    Yongmeng Wu & Jinyi Zhi

  2. Queen Mary University of London, Mile End Road, E1 4NS, London, UK

    Nick Bryan-Kinns

Authors
  1. Yongmeng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nick Bryan-Kinns
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinyi Zhi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongmeng Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Bryan-Kinns, N. & Zhi, J. Exploring visual stimuli as a support for novices’ creative engagement with digital musical interfaces. J Multimodal User Interfaces 16, 343–356 (2022). https://doi.org/10.1007/s12193-022-00393-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-022-00393-3

Keywords

Search

Navigation