Abstract
This paper draws upon the theory of embodied cognition to provide a robust account of how gestural interactions with and around multi-touch tabletops can play an important role in facilitating collaborative meaning-making, particularly in the context of science data visualizations. Embodied cognition is a theory of learning that implies that thinking and perception are shaped by interactions with the physical environment. Previous research has used embodied cognition as a theoretical framework to inform the design of large touchscreen learning applications such as for multi-touch tabletops. However, this prior work has primarily assumed that learning is occurring during any motion or interaction, without considering how specific interactions may be linked to particular instances of collaborative learning supported by embodiment. We investigate this question in the context of collaborative learning from data visualizations of global phenomena such as ocean temperatures. We followed a user-centered, iterative design approach to build a tabletop prototype that facilitated collaborative meaning-making and used this prototype as a testbed in a laboratory study with 11 family groups. We qualitatively analyzed learner groups’ co-occurring utterances and gestures to identify the nature of gestural interactions groups used when their utterances signaled the occurrence of embodiment during collaborative meaning-making. Our findings present an analysis of both touchscreen and above-the-surface gestural interactions that were associated with instances of embodied cognition. We identified four types of gestural interactions that promote scientific discussion and collaborative meaning-making through embodied cognition: (T1) gestures for orienting the group; (T2) cooperative gestures for facilitating group meaning-making; (T3) individual intentional gestures for facilitating group meaning-making; and (T4) gestures for articulating conceptual understanding to the group. Our work illustrates interaction design opportunities for affording embodied cognition and will inform the design of future interactive tabletop experiences in the domain of science learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abrahamson, D., Andrade, A., Bakker, A., Nathan, M. J., Walkington, C., Lindgren, R., … Lindwall, O. (2018). Moving forward: In search of synergy across diverse views on the role of physical movement in design for STEM education. Proceedings of international conference of the learning sciences, 2, 1243–1250.
Abrahamson, D., & Lindgren, R. (2014). Embodiment and embodied design. In In R. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences. https://doi.org/10.1017/CBO9781139519526.022.
Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners’ and teachers’ gestures. Journal of the Learning Sciences, 21(2), 247–286. https://doi.org/10.1080/10508406.2011.611446.
Anthony, L., Stofer, K. A., Luc, A., & Wobbrock, J. O. (2016). Gestures by children and adults on touch tables and touch walls in a public science center. Proceedings of the Conference on Interaction Design and Children, 344–355. https://doi.org/10.1145/2930674.2930682.
Anthony, L., Yang, J., & Koedinger, K. R. (2012). A paradigm for handwriting-based intelligent tutors. International Journal of Human-Computer Studies, 70(11), 866–887. https://doi.org/10.1016/j.ijhcs.2012.04.003.
Baccaglini-Frank, A., & Maracci, M. (2015). Multi-touch technology and preschoolers’ development of number sense. Digital Experiences in Mathematics Education, 1, 7–27. https://doi.org/10.1007/s40751-015-0002-4.
Becvar, L. A., Hollan, J., & Hutchins, E. (2005). Hands as molecules: Representational gestures used for developing theory in a scientific laboratory. Semiotica, 2005(156), 89–112. https://doi.org/10.1515/semi.2005.2005.156.89.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa.
Cheek, K. A. (2010). Commentary: A summary and analysis of twenty-seven years of geoscience conceptions research. Journal of Geoscience Education, 58(3), 122–134. https://doi.org/10.5408/1.3544294.
Danish, J. A., Enyedy, N., Saleh, A., & Humburg, M. (2020). Learning in embodied activity framework: A sociocultural framework for embodied cognition. International Journal of Computer-Supported Collaborative Learning, 15, 49–87. https://doi.org/10.1007/s11412-020-09317-3.
Dasu, K., Ma, K.-L., Ma, J., & Frazier, J. (2019). Sea of Genes: Combining Animation and Narrative Strategies to Visualize Metagenomic Data for Museums. Retrieved June 10, 2020, from https://arxiv.org/abs/1906.01071
Davidsen, J., & Ryberg, T. (2017). “This is the size of one meter”: Children’s bodily-material collaboration. International Journal of Computer-Supported Collaborative Learning, 12(1), 65–90. https://doi.org/10.22318/cscl2015.150.
Devine-Wright, P. (2013). Think global, act local? The relevance of place attachments and place identities in a climate changed world. Global Environmental Change, 23(1), 61–69. https://doi.org/10.1016/j.gloenvcha.2012.08.003.
Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and computational approaches (pp. 1–19). Oxford: Elsevier.
Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In Technology-Enhanced Learning (pp. 3–19). https://doi.org/10.1007/978-1-4020-9827-7_1.
Dubé, A. K., & McEwen, R. N. (2015). Do gestures matter? The implications of using touchscreen devices in mathematics instruction. Learning and Instruction, 40, 89–98. https://doi.org/10.1016/j.learninstruc.2015.09.002.
Epps, J., Oviatt, S., & Chen, F. (2004). Integration of speech and gesture inputs during multimodal interaction. Proceedings of the Australian Conference on Human-Computer Interaction.
Fleck, R., Rogers, Y., Yuill, N., Marshall, P., Carr, A., Rick, J., & Bonnett, V. (2009). Actions speak loudly with words: Unpacking collaboration around the table. Proceedings of the Conference on Interactive Tabletops and Surfaces, 189–196. https://doi.org/10.1145/1731903.1731939.
Geller, T. (2006). Interactive tabletop exhibits in museums and galleries. IEEE Computer Graphics and Applications, 26(5), 6–11. https://doi.org/10.1109/MCG.2006.111.
Goldin-Meadow, S., & Beilock, S. L. (2010). Action’s influence on thought: The case of gesture. Perspectives on Psychological Science, 5(6), 664–674. https://doi.org/10.1177/1745691610388764.
Goldinger, S. D., Papesh, M. H., Barnhart, A. S., Hansen, W. A., & Hout, M. C. (2016). The poverty of embodied cognition. Psychonomic Bulletin and Review, 23(4), 959–978. https://doi.org/10.3758/s13423-015-0860-1.
Goldman, K. H., & Gonzalez, J. (2014). Open exhibits multitouch table use findings. Ideum Open Exhibits Papers. Retrieved from http://openexhibits.org/exhibit-design/multitouch-table-research-findings/8801/
Goldman, K. H., Kessler, C., & Danter, E. (2010). Science On a Sphere: Cross-site Summative Evaluation. In Institute for Learning Innovation. Retrieved from http://www.oesd.noaa.gov/network/SOS_evals/SOS_Final_Summative_Report.pdf
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. https://doi.org/10.1016/S0378-2166(99)00096-X.
Greenberg, S., Carpendale, S., Marquardt, N., & Buxton, B. (2011). Sketching user experiences: The workbook. Elsevier.
Gutwill, J. P., & Allen, S. (2010). Facilitating family group inquiry at science museum exhibits. Science Education, 94(4), 710–742. https://doi.org/10.1002/sce.20387.
Hoadley, C. P. (2002). Creating context: Design-based research in creating and understanding CSCL. Proceedings of the Conference on Computer Support for Collaborative Learning, 453–462. https://doi.org/10.3115/1658616.1658679.
Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Toward a new Foundation for human-computer interaction research. Proceedings of Transactions on Computer-Human Interaction (TOCHI’00), 7(2), 174–196. https://doi.org/10.1145/353485.353487.
Horn, M., Leong, Z. A., Block, F., Diamond, J., Evans, E. M., Phillips, B., & Shen, C. (2012). Of BATs and APEs: An Interactive Tabletop Game for Natural History Museums. Proceedings of the Conference on Human Factors in Computing Systems, 2059–2068. https://doi.org/10.1145/2207676.2208355.
Hornecker, E. (2005). A design theme for tangible interaction: Embodied facilitation. Proceedings of the European Conference on Computer Supported Cooperative Work, 23–43. https://doi.org/10.5555/1242029.1242031.
Hornecker, E. (2008). “I Don’t understand it either, but it is cool” - visitor interactions with a multi-touch table in a museum. IEEE Workshop on Horizontal Interactive Human Computer Systems, 113–120. https://doi.org/10.1109/TABLETOP.2008.4660193.
Hullman, J., Adar, E., & Shah, P. (2011). Benefitting InfoVis with visual difficulties. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2213–2222. https://doi.org/10.1109/TVCG.2011.175.
Kharrufa, A. S., & Olivier, P. (2010). Exploring the requirements of tabletop interfaces for education. International Journal of Learning Technology, 5(1), 42–62.
Kim, K. T., & Elmqvist, N. (2012). Embodied lenses for collaborative visual queries on tabletop displays. Information Visualization, 11(4), 319–338. https://doi.org/10.1177/1473871612441874.
Kirschner, P. A., Sweller, J., Kirschner, F., Zambrano, R., & J. (2018). From cognitive load theory to collaborative cognitive load theory. International Journal of Computer-Supported Collaborative Learning, 13(2), 213–233. https://doi.org/10.1007/s11412-018-9277-y.
Kirsh, D. (2013). Embodied cognition and the magical future of interaction design. ACM Transactions on Computer-Human Interaction, 20(1), 30 pages. https://doi.org/10.1145/2442106.2442109, 1, 30.
Kiverstein, J. D., & Rietveld, E. (2018). Reconceiving representation-hungry cognition: An ecological-enactive proposal. Adaptive Behavior, 26(4), 147–163. https://doi.org/10.1177/1059712318772778.
Köpsel, A., & Bubalo, N. (2015). Benefiting from legacy Bias. Interactions, 22(5), 44–47. https://doi.org/10.1145/2803169.
Lakoff, G., & Johnson, M. (2003). Metaphors we live by. Chicago: University of Chicago Press.
Lin, M., Preston, A., Kharrufa, A., & Kong, Z. (2016). Making L2 learners’ reasoning skills visible: The potential of computer supported collaborative learning environments. Thinking Skills and Creativity, 22, 303–322. https://doi.org/10.1016/j.tsc.2016.06.004.
Lobben, A., & Lawrence, M. (2015). Synthesized model of geospatial thinking. The Professional Geographer, 67(3), 307–318. https://doi.org/10.1080/00330124.2014.935155.
Ma, J. (2008). Visitors’ Interpretations of Images of the Nanoscale. In Nanoscale Informal Science Education Network. Retrieved from www.nisenet.org/sites/default/catalog/eval/uploads/2009/04/1353/visitor_interpretations.pdf.
Ma, J., Liao, I., Ma, K. L., & Frazier, J. (2012). Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2799–2808. https://doi.org/10.1109/TVCG.2012.244.
McNeill, D. (1992). Hand and mind : What gestures reveal about thought. University of Chicago Press.
Morris, M. R., Huang, A., Paepcke, A., & Winograd, T. (2006). Cooperative gestures: Multi-user gestural interactions for co-located groupware. Proceedings of the Conference on Human Factors in Computing Systems, 1201–1210. https://doi.org/10.1145/1124772.1124952.
Oviatt, S. (1997). Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 12(1–2), 93–129. https://doi.org/10.1080/07370024.1997.9667241.
Piper, A. M., Friedman, W., & Hollan, J. D. (2012). Setting the stage for embodied activity: Scientific discussion around a multi-touch tabletop display. International Journal of Learning Technology, 7(1), 58–78. https://doi.org/10.1504/IJLT.2012.046866.
Piper, A. M., & Hollan, J. D. (2009). Tabletop displays for small group study: Affordances of paper and digital materials. Proceedings of the Conference on Human Factors in Computing Systems, 1227–1236. https://doi.org/10.1145/1518701.1518885.
Price, S., Falcão, T. P., Sheridan, J. G., & Roussos, G. (2009). The effect of representation location on interaction in a tangible learning environment. Proceedings of the Conference on Tangible and Embedded Interaction, 85–92. https://doi.org/10.1145/1517664.1517689.
Rick, J., Marshall, P., & Yuill, N. (2011). Beyond one-size-fits-all: How interactive tabletops support collaborative learning. Proceedings of the Conference on Interaction Design and Children, 109–117. https://doi.org/10.1145/1999030.1999043.
Rogers, Y., & Muller, H. (2006). A framework for designing sensor-based interactions to promote exploration and reflection in play. International Journal of Human Computer Studies, 64(1), 1–14. https://doi.org/10.1016/j.ijhcs.2005.05.004.
Rust, K., Malu, M., Anthony, L., & Findlater, L. (2014). Understanding child-defined gestures and Children’s mental models for touchscreen tabletop interaction. Proceedings of the Conference on Interaction Design and Children, 201–204. https://doi.org/10.1145/2593968.2610452.
Schuman, C., Stofer, K. A., Anthony, L., Neff, H., Chang, P., Soni, N., et al. (2020). Ocean data visualization on a Touchtable demonstrates group content learning, science practices use, and potential embodied cognition. Research in Science Education, 1–13. https://doi.org/10.1007/s11165-020-09951-9.
Segal, A. (2011). Do gestural interfaces promote thinking? Embodied Interaction: Congruent Gestures and Direct Touch Promote Performance in Math. Columbia University.
Sinclair, N., & Pimm, D. (2015). Mathematics using multiple senses: Developing finger gnosis with three- and four-year-olds in an era of multi-touch technologies. Asia-Pacific Journal of Research in Early Childhood Education, 9, 99–110. https://doi.org/10.17206/apjrece.2015.9.3.99.
Soni, N., & Anthony, L. (2019). HCI methodologies for designing natural user interactions that do not interfere with learning. Paper for the “making the learning sciences count: Impacting Association for Computing Machinery Communities in human-computer interaction” workshop, International Conference of Computer-Supported Collaborative Learning, 5 pages.
Soni, N., Darrow, A., Luc, A., Gleaves, S., Schuman, C., Neff, H., … Anthony, L. (2019b). Analysis of touchscreen interactive gestures during embodied cognition in collaborative tabletop science learning experiences. Proceedings of the Conference of Computer Supported Collaborative Learning, 9–16.
Soni, N., Gleaves, S., Neff, H., Morrison-Smith, S., Esmaeili, S., Mayne, I., et al. (2019c). Do user-defined gestures for Flatscreens generalize to interactive spherical displays for adults and children? Proceedings of the ACM International Symposium on Pervasive Displays, 7. https://doi.org/10.1145/3321335.3324941.
Soni, N., Gleaves, S., Neff, H., Morrison-Smith, S., Esmaeili, S., Mayne, I., et al. (2020). Adults’ and Children’s mental models for gestural interactions with interactive spherical displays. Proceedings of the Conference on Human Factors in Computing Systems, 1–12. https://doi.org/10.1145/3313831.3376468.
Stahl, G. (2018). A Theory of Group Cognition in CSCL. Retrieved from http://gerrystahl.net/pub/theory.pdf
Steier, R., Kersting, M., & Silseth, K. (2019). Imagining with improvised representations in CSCL environments. International Journal of Computer-Supported Collaborative Learning, 14, 109–136. https://doi.org/10.1007/s11412-019-09295-1.
Stofer, K. A. (2016). When a picture Isn’t worth 1000 words: Learners struggle to find meaning in data visualizations. Journal of Geoscience Education, 64(3), 231–241. https://doi.org/10.5408/14-053.1.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
Tang, J. C. (1991). Findings from observational studies of collaborative work. International Journal of Man-Machine Studies, 34(2), 143–160. https://doi.org/10.1016/0020-7373(91)90039-A.
Tse, E., Shen, C., Greenberg, S., & Forlines, C. (2006). Enabling interaction with single user applications through speech and gestures on a multi-user tabletop. Proceedings of the Conference on Advanced Visual Interfaces, 336–343. https://doi.org/10.1145/1133265.1133336.
Vygotsky, L. (1986). Thought and language. Cambridge Mass: MIT Press.
Vygotsky, L. S. (1978). Mind in society: Development of higher psychological processes. Harvard University Press.
Wee, L. (2005). Class-inclusion and correspondence models as discourse types: A framework for approaching metaphorical discourse. Language in Society, 34(2), 219–238. https://doi.org/10.1017/S0047404505050098.
Wilson, A. D., & Golonka, S. (2013). Embodied Cognition is Not What You Think It Is. Frontiers in Psychology, 4, 58 pages. https://doi.org/10.3389/fpsyg.2013.00058.
Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin and Review, 9, 625–636. https://doi.org/10.3758/BF03196322.
Xambó, A., Jewitt, C., & Price, S. (2014). Towards an integrated methodological framework for understanding embodiment in HCI. Proceedings of the Conference on Human Factors in Computing Systems, 1411–1416. https://doi.org/10.1145/2559206.2581276.
Zheng, L., Cui, P., & Zhang, X. (2020). Does collaborative learning design align with enactment? An innovative method of evaluating the alignment in the CSCL context. International Journal of Computer-Supported Collaborative Learning, 15, 193–226. https://doi.org/10.1007/s11412-020-09320-8.
Acknowledgments
This work is partially supported by National Science Foundation Grant Awards #DRL-1612485 and #IIS-1552598. Any opinions, findings, and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect these agencies’ views. The authors also thank the Florida Museum of Natural History for allowing us to recruit participants from their visitors; our advisory board members, Betty Dunckel and Julia Plummer, for contributing to our thinking related to embodied cognition; and all our pilot participants. A preliminary version of the analysis in this paper that included only T1 and T2 was presented at CSCL 2019 (Soni et al. 2019b). This work was conducted while author Jeremy Alexandre was a summer intern at the University of Florida.
Availability of data and material
The authors plan to make the anonymized study data publicly available on the open science framework data repository in the future: https://osf.io/rjzkh/
Code availability
The authors plan to make the codebase for the multi-touch tabletop application used in the study publicly available on the open science framework data repository in the future: https://osf.io/rjzkh/
Funding
This work is partially supported by National Science Foundation Grant Awards #DRL-1612485 and #IIS-1552598.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest/competing interests
Not Applicable.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Soni, N., Darrow, A., Luc, A. et al. Affording embodied cognition through touchscreen and above-the-surface gestures during collaborative tabletop science learning. Intern. J. Comput.-Support. Collab. Learn 16, 105–144 (2021). https://doi.org/10.1007/s11412-021-09341-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11412-021-09341-x