Skip to main content
SpringerLink
Log in

Interactive exploration of a hierarchical spider web structure with sound

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

3D spider webs exhibit highly intricate fiber architectures and owe their outstanding performance to a hierarchical organization that spans orders of magnitude in length scale from the molecular silk protein, to micrometer-sized fibers, and up to cm-scale web. Similarly, but in a completely different physical manifestation, music has a hierarchical structure composed of elementary sine wave building blocks that can be combined with other waveforms to create complex timbres, which are then arranged within larger-scale musical compositions. Although apparently different, spider webs and music have many similarities, as we point out in this work. Here, we propose an intuitive and interactive way to explore and visualize a 3D Cyrtophora citricola spider web geometry that has been digitally modeled with micron-scale details from full-scale laboratory experiments. We use model-based sonification to translate the web architecture into sound, allowing for aural perception and interpretation of its essential topological features. We implement this sonification using Unity3D and Max/MSP to create an interactive spider web environment in which a user travels through a virtual spider web. Each silk fiber in their field of view is sonified using different sine waves. Together, the sonified fibers create new and more complex timbres that reflects the architecture of 3D spider webs. These concepts are implemented into a spider web-based instrument for live performances, art installations and data exploration. It provides an unprecedented and creative way to immerse the composer, audience and user in an immersive multimedia experience generated by the complexity of a 3D spider web.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and material

The spider web images and model are available from the corresponding author on reasonable request.

Code availability

The sonification model (Unity3D game and Max/MSP patch) are available from the corresponding author on reasonable request.

References

  1. Su I, Qin Z, Bisshop A et al (2021) Sonification of a 3-D spider web and reconstitution for musical composition using granular synthesis. Comput Music J (in Press)

  2. Hermann T, Hunt A, Neuhoff JG (2011) The sonification handbook. Logos Verlag, Berlin

    Google Scholar 

  3. Bovermann T, Hermann T, Ritter H (2006) Tangible data scanning sonification model. Georgia Institute of Technology, Atlanta

    Google Scholar 

  4. Kramer G, Walker BN, Bonebright T et al (1999) The sonification report: status of the field and research agenda. Report prepared for the national science foundation by members of the international community for auditory display. Int Community Audit Disp (ICAD), St Fe, NM

  5. Pauletto S, Hunt A (2009) Interactive sonification of complex data. Int J Hum Comput Stud 67:923–933. https://doi.org/10.1016/j.ijhcs.2009.05.006

    Article  Google Scholar 

  6. Idreos S, Papaemmanouil O, Chaudhuri S (2015) Overview of data exploration techniques. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 277–281

  7. Papachristodoulou P, Betella A, Verschure P (2014) Sonification of large datasets in a 3D immersive environment: a neuroscience case study. In: ACHI2014: the seventh international conference on advances in computer-human interactions, pp 35–40

  8. Yu CH, Qin Z, Martin-Martinez F, Buehler MJ (2019) A self-consistent sonification method to translate amino acid sequences into musical compositions and application in protein design using AI. ACS Nano 13:7471–7482

    Article  Google Scholar 

  9. Jeon M, Smith MT, Walker JW, Kuhl SA (2014) Constructing the immersive interactive sonification platform (iISoP). In: Streitz N, Markopoulos P (eds) Distributed, ambient, and pervasive interactions. Springer, Cham, pp 337–348

    Chapter  Google Scholar 

  10. Barrett N (2016) Interactive spatial sonification of multidimensional data for composition and auditory display. Comput Music J 40:47–69. https://doi.org/10.1162/COMJ_a_00358

    Article  Google Scholar 

  11. Gu GX, Su I, Sharma S et al (2016) Three-dimensional-printing of bio-inspired composites. J Biomech Eng 138:21006

    Article  Google Scholar 

  12. Su I, Buehler MJ (2016) Spider silk: dynamic mechanics. Nat Mater 15:1054–1055. https://doi.org/10.1038/nmat4721

    Article  Google Scholar 

  13. Su I, Buehler MJ (2016) Nanomechanics of silk: the fundamentals of a strong, tough and versatile material. Nanotechnology 27:302001. https://doi.org/10.1088/0957-4484/27/30/302001

    Article  Google Scholar 

  14. Su I, Jung GS, Narayanan N, Buehler MJ (2020) Perspectives on three-dimensional printing of self-assembling materials and structures. Curr Opin Biomed Eng 15:59–67. https://doi.org/10.1016/j.cobme.2020.01.003

    Article  Google Scholar 

  15. Giesa T, Spivak D, Buehler M (2011) Reoccurring patterns in hierarchical protein materials and music: the power of analogies. Bionanoscience 1:153

    Article  Google Scholar 

  16. Wong JY, McDonald J, Taylor-Pinney M et al (2012) Materials by design: merging proteins and music. Nano Today 7:488–495. https://doi.org/10.1016/j.nantod.2012.09.001

    Article  Google Scholar 

  17. Mortimer B (2017) Biotremology: do physical constraints limit the propagation of vibrational information? Anim Behav 130:165–174. https://doi.org/10.1016/j.anbehav.2017.06.015

    Article  Google Scholar 

  18. Hill PSM, Wessel A (2016) Biotremology. Curr Biol 26:R187–R191

    Article  Google Scholar 

  19. Mortimer B, Soler A, Siviour CR et al (2016) Tuning the instrument: sonic properties in the spider’s web. J R Soc Interface. https://doi.org/10.1098/rsif.2016.0341

    Article  Google Scholar 

  20. Mortimer B, Gordon S, Drodge D et al (2013) Sonic properties of silks. XIV Int Conf Invertebr Sound Vib ISV 2013:43

    Google Scholar 

  21. Saraceno T, Bisshop A, Krell A, Mühlethaler R (2019) Arachnid orchestras: artistic research in vibrational interspecies communication. In: Biotremology: studying vibrational behavior. Springer, pp 485–509

  22. Su I, Qin Z, Saraceno T et al (2018) Imaging and analysis of a threedimensional spider web architecture. J R Soc Interface. https://doi.org/10.1098/rsif.2018.0193

    Article  Google Scholar 

  23. Wulff C (2010) Zu Hause bei der Schwarzen Witwe oder das letzte Projekt mit Rolf-Dieter Düppe. Festschrift anlässlich der Pensionierung von Dr-Ing Rolf-Dieter Düppe nach 36 Jahren am Inst für Photogramm und Kartographie; Schriftenr/Fachrichtung Geodäsie 30:8

  24. Luhmann T (2017) Nahbereichsphotogrammetrie, Errata. Wichmann Verlag, Heidelberg

    Google Scholar 

  25. Luhmann T (2010) 3D-Rekonstruktion von Spinnennetzen. Wichmann Verlag, Heidelberg

    Google Scholar 

  26. Papachristodoulou P, Betella A, Manzolli J, Verschure PFMJ (2015) Augmenting the navigation of complex data sets using sonification: a case study with BrainX3 Panagiota. In: 2015 IEEE 2nd VR Workshop on sonic interactions for virtual environments, SIVE 2015—proceedings. Institute of Electrical and Electronics Engineers Inc., pp 3–8

  27. Rossing TD, SpringerLink (2010) The science of string instruments. [electronic resource]. Springer, New York, 2010

  28. Bennett WR, Holland CK (2018) The science of musical sound. Springer, Berlin

    Book  Google Scholar 

  29. Robinson DW, Dadson RS (1956) A re-determination of the equal-loudness relations for pure tones. Br J Appl Phys 7:166–181. https://doi.org/10.1088/0508-3443/7/5/302

    Article  Google Scholar 

  30. Chadabe J (1984) Interactive composing: an overview. Comput Music J. https://doi.org/10.2307/3679894

    Article  Google Scholar 

  31. Engelmann S (2016) Social spiders and hybrid webs at Studio Tomás Saraceno. Cult Geogr 24:161–169. https://doi.org/10.1177/1474474016647371

    Article  Google Scholar 

  32. Vets T, Nijs L, Lesaffre M et al (2017) Gamified music improvisation with BilliArT: a multimodal installation with balls. J Multimodal User Interfaces 11:25–38. https://doi.org/10.1007/s12193-016-0224-9

    Article  Google Scholar 

  33. MATLAB (2019) 9.6.0.1099231 (R2019a). The MathWorks Inc., Natick, Massachusetts

  34. Sporns O (2013) The human connectome: origins and challenges. Neuroimage 80:53–61. https://doi.org/10.1016/j.neuroimage.2013.03.023

    Article  Google Scholar 

  35. Lindborg P (2018) Interactive sonification of weather data for the locust wrath, a multimedia dance performance. Leonardo 51:466–474

    Article  Google Scholar 

  36. Haixia Z, Smith BK, Norman K et al (2005) Interactive sonification of choropleth maps. IEEE Multimed 12:26–35. https://doi.org/10.1109/MMUL.2005.28

    Article  Google Scholar 

  37. Zotkin DYN, Hwang J, Duraiswaini R, Davis LS (2003) HRTF personalization using anthropometric measurements. In: 2003 IEEE workshop on applications of signal processing to audio and acoustics (IEEE Cat. No.03TH8684). pp 157–160

Download references

Funding

This research was supported by the MIT Center for Art, Science and Technology (CAST) program, The Mellon Foundation, and Studio Tomás Saraceno GmbH. We are grateful for the inspiration from Studio Saraceno to develop this instrument for a public performance in the context of his exhibition ON AIR at the Palais de Tokyo, Paris in November 2018, supported by Festival d'Automne à Paris and curated by Rebecca Lamarche-Vadel. Additional support provided by the Office of Naval Research (grant # N00014-16-1-2333), the National Institutes of Health (NIH) U01EB014976, and The Mellon Foundation.

Author information

Authors and Affiliations

  1. Laboratory for Atomistic and Molecular Mechanics (LAMM), Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, 77 Massachusetts Ave., Cambridge, MA, 02139, USA

    Isabelle Su & Markus J. Buehler

  2. Department of Music and Theater Arts, Massachusetts Institute of Technology, 77 Massachusetts Ave., Cambridge, MA, 02139, USA

    Ian Hattwick, Christine Southworth & Evan Ziporyn

  3. Studio Tomás Saraceno, Hauptstrasse 11/12, 10317, Lichtenberg, Berlin, Germany

    Ally Bisshop, Roland Mühlethaler & Tomás Saraceno

  4. Center for Computational Science and Engineering, Schwarzman College of Computing, Massachusetts Institute of Technology, 77 Massachusetts Ave., Cambridge, MA, 02139, USA

    Markus J. Buehler

Authors
  1. Isabelle Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ian Hattwick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christine Southworth
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Evan Ziporyn
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ally Bisshop
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Roland Mühlethaler
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tomás Saraceno
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Markus J. Buehler
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

IS, IH, CS, EZ, TS, MJB designed the research and overall project. IS created the spider web model, the Unity3D game, virtual reality, and Max/MSP patch for the real-time interactive sonification model. IS, IH, CS, EZ and MJB designed the sonification model. IH created the Max/MSP patch for real-time sound sculpting of the sonification of the spider web. CS and EZ designed the Spider’s Canvas/Arachnodrone physical installation and performance. CS collected photographs of spiders and webs and created a photo montage video superposed onto the spider web model projections. CS designed the visuals for the performance. IS, IH, CS, EZ performed together the Spider’s Canvas/Arachnodrone. During the performances: IS controlled player’s point of view in the virtual spider web and fiber esthetics, IH sound-sculpted in real-time web sonification, CS performed on the Ebow and EZ performed the EWI. TS commissioned the premiere of the Spider’s Canvas/Arachnodrone. AB and RM edided the paper. IS, IH, CS, EZ and MJB wrote the paper.

Corresponding author

Correspondence to Markus J. Buehler.

Ethics declarations

Conflict of interest

We declare we have no conflicts of interest and competing interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 105 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, I., Hattwick, I., Southworth, C. et al. Interactive exploration of a hierarchical spider web structure with sound. J Multimodal User Interfaces 16, 71–85 (2022). https://doi.org/10.1007/s12193-021-00375-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-021-00375-x

Keywords

Search

Navigation