Abstract
While observing gesture has been shown to benefit narrative recall and learning, research has yet to show whether gestures that provide information that is missing from speech benefit narrative recall. This study explored whether observing gestures that relay the same information as speech and gestures that provide information missing from speech differentially affect narrative recall in university students. Participants were presented with a videotaped narrative told in one of four conditions: with gestures and no missing verbal information, with gestures and missing verbal information, with no gestures and no missing verbal information, or with no gestures and missing verbal information. Results showed that observing gestures that provided additional information to speech (i.e., when the speech was missing vital information) enhanced narrative recall compared to observing no gestures, while observing gestures that did not provide additional information to speech were no more beneficial than observing no gestures at all. Findings from the current study provide valuable insight into the beneficial effect of iconic gesture on narrative recall, with important implications for education and learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Supplied on request by contacting the corresponding author, Nicole Dargue.
References
Austin, E. E., & Sweller, N. (2014). Presentation and production: The role of gesture in spatial communication. Journal of Experimental Child Psychology, 122, 92–103. https://doi.org/10.1016/j.jecp.2013.12.008
Austin, E. E., & Sweller, N. (2017). Getting to the elephants: Gesture and preschoolers’ comprehension of route direction information. Journal of Experimental Child Psychology, 163, 1–14. https://doi.org/10.1016/j.jecp.2017.05.016
Austin, E. E., Sweller, N., & Van Bergen, P. (2018). Pointing the way forward: Gesture and adults’ recall of route direction information. Journal of Experimental Psychology: Applied, 24(4), 490–508. https://doi.org/10.1037/xap0000168
Biau, E., & Soto-Faraco, S. (2013). Beat gestures modulate auditory integration in speech perception. Brain and Language, 124(2), 143–152. https://doi.org/10.1016/j.bandl.2012.10.008
Cartmill, E. A., Ece Demir, Ö., & Goldin-Meadow, S. (2012). Studying gesture. In E. Hoff (Ed.), Research methods in child language (pp. 208–225). Wiley. https://doi.org/10.1002/9781444344035.ch14
Cassell, J., McNeill, D., & McCullough, K.-E. (1994). Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics and Cognition, 7(1), 1–34. https://doi.org/10.1075/pc.7.1.03cas
Church, R., Garber, P., & Rogalski, K. (2007). The role of gesture in memory and social communication. Gesture, 7(2), 137–158. https://doi.org/10.1075/gest.7.2.02bre
Cook, S. W. (2018). Enhancing learning with hand gestures: Potential mechanisms. Psychology of learning and motivation—Advances in research and theory (pp. 107–133). Elsevier. https://doi.org/10.1016/bs.plm.2018.10.001
Dargue, N., & Sweller, N. (2018a). Not all gestures are created equal: The effects of typical and atypical iconic gestures on narrative comprehension. Journal of Nonverbal Behavior, 42(3), 327–345. https://doi.org/10.1007/s10919-018-0278-3
Dargue, N., & Sweller, N. (2018b). Donald Duck’s garden: The effects of observing iconic reinforcing and contradictory gestures on narrative comprehension. Journal of Experimental Child Psychology, 175, 96–107. https://doi.org/10.1016/j.jecp.2018.06.004
Dargue, N., & Sweller, N. (2019). Learning stories through gesture: Gesture’s effects on child and adult narrative comprehension. Educational Psychology Review, 32(1), 249–276. https://doi.org/10.1007/s10648-019-09505-0
Dargue, N., & Sweller, N. (2020). Two hands and a tale: When gestures benefit adult narrative comprehension. Learning and Instruction, 68, 101331. https://doi.org/10.1016/j.learninstruc.2020.101331
Dargue, N., Sweller, N., & Jones, M. P. (2019). When our hands help us understand: A meta-analysis into the effects of gesture on comprehension. Psychological Bulletin, 145(8), 765–784. https://doi.org/10.1037/bul0000202
Dijkstra, K., & Zwaan, R. A. (2014). Memory and Action. In Shapiro, L. (Ed.), The Routledge handbook of embodied cognition (pp. 296–305). Taylor & Francis Group. https://doi.org/10.4324/9781315775845
Driskell, J. E., & Radtke, P. H. (2003). The effect of gesture on speech production and comprehension. Human Factors: THe Journal of the Human Factors and Ergonomics Society, 45(3), 445–454. https://doi.org/10.1518/hfes.45.3.445.27258
Dunn, L. M., & Dunn, D. M. (2007). Peabody picture vocabulary test-fourth edition. PsycTESTS. https://doi.org/10.1037/t15144-000
Feyereisen, P. (2006). Further investigation on the mnemonic effect of gestures: Their meaning matters. European Journal of Cognitive Psychology, 18(2), 185–205. https://doi.org/10.1080/09541440540000158
Flevares, L. M., & Perry, M. (2001). How many do you see? The use of nonspoken representations in first-grade mathematics lessons. Journal of Educational Psychology, 93(2), 330–345. https://doi.org/10.1037/0022-0663.93.2.330
Fronda, G., & Balconi, M. (2020). The effect of interbrain synchronization in gesture observation: A fNIRS study. Brain and Behavior. https://doi.org/10.1002/brb3.1663
Goldin-Meadow, S., Kim, S., & Singer, M. (1999). What the teacher's hands tell the student's mind about math. Journal of Educational Psychology, 91(4), 720–730. https://doi.org/10.1037/0022-0663.91.4.720
Hauk, O., Johnsrude, I., & Pulvermüller, F. (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41(2), 301–307. https://doi.org/10.1016/S0896-6273(03)00838-9
Holle, H., Gunter, T. C., Rüschemeyer, S. A., Hennenlotter, A., & Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. NeuroImage, 39(4), 2010–2024. https://doi.org/10.1016/j.neuroimage.2007.10.055
Holle, H., Obermeier, C., Schmidt-Kassow, M., Friederici, A. D., Ward, J., & Gunter, T. C. (2012). Gesture facilitates the syntactic analysis of speech. Frontiers in Psychology, 3, 1–12. https://doi.org/10.3389/fpsyg.2012.00074
Hostetter, A. B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137(2), 297–315. https://doi.org/10.1037/a0022128
Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review, 15, 495–514. https://doi.org/10.3758/PBR.15.3.495
Hostetter, A. B., & Alibali, M. W. (2010). Language, gesture, action! A test of the gesture as simulated action framework. Journal of Memory and Language, 63(2), 245–257. https://doi.org/10.1016/j.jml.2010.04.003
Hostetter, A. B., & Alibali, M. W. (2019). Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin and Review, 26(3), 721–752. https://doi.org/10.3758/s13423-018-1548-0
Hubbard, A. L., Wilson, S. M., Callan, D. E., & Dapretto, M. (2009). Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Human Brain Mapping, 30(3), 1028–1037. https://doi.org/10.1002/hbm.20565
Hupp, J. M., & Gingras, M. C. (2016). The role of gesture meaningfulness in word learning. Gesture, 15(3), 340–356. https://doi.org/10.1075/gest.15.3.04hup
Ianì, F., & Bucciarelli, M. (2017). Mechanisms underlying the beneficial effect of a speaker’s gestures on the listener. Journal of Memory and Language, 96, 110–121. https://doi.org/10.1016/j.jml.2017.05.004
Ianì, F., & Bucciarelli, M. (2018). Relevance of the listener’s motor system in recalling phrases enacted by the speaker. Memory, 26(8), 1084–1092. https://doi.org/10.1080/09658211.2018.1433214
Ianì, F., Burin, D., Salatino, A., Pia, L., Ricci, R., & Bucciarelli, M. (2018). The beneficial effect of a speaker’s gestures on the listener’s memory for action phrases: The pivotal role of the listener’s premotor cortex. Brain and Language, 180–182, 8–13. https://doi.org/10.1016/j.bandl.2018.03.001
Igualada, A., Esteve-Gibert, N., & Prieto, P. (2017). Beat gestures improve word recall in 3- to 5-year-old children. Journal of Experimental Child Psychology, 156, 99–112. https://doi.org/10.1016/j.jecp.2016.11.017
Joshi, R. (2005). Vocabulary: A critical component of comprehension. Reading and Writing Quarterly, 21(3), 209–219. https://doi.org/10.1080/10573560590949278
Kelly, S. D. (2001). Broadening the units of analysis in communication: Speech and nonverbal behaviours in pragmatic comprehension. Journal of Child Language, 28(2), 325–349. https://doi.org/10.1017/S0305000901004664
Kelly, S. D., Barr, D. J., Church, R., & Lynch, K. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40(4), 577–592. https://doi.org/10.1006/jmla.1999.2634
Kelly, S. D., & Church, R. B. (1998). A comparison between children’s and adults’ ability to detect conceptual information conveyed through representational gestures. Child Development, 69(1), 85–93. https://doi.org/10.1111/j.1467-8624.1998.tb06135.x
Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95(2), 163–182. https://doi.org/10.1037/0033-295X.95.2.163
Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57(3), 396–414. https://doi.org/10.1016/j.jml.2007.06.005
Levelt, W. J. M. (1989). Speaking: From intention to articulation. MIT Press.
Littlejohn, S. W., & Foss, K. A. (2010). Theories of human communication (10th ed.). Waveland Press Inc.
Lynch, J. S., van den Broek, P., Kremer, K. E., Kendeou, P., White, M. J., & Lorch, E. P. (2008). The development of narrative recall and its relation to other early reading skills. Reading Psychology, 29(4), 327–365. https://doi.org/10.1080/02702710802165416
Macoun, A., & Sweller, N. (2016). Listening and watching: The effects of observing gesture on preschoolers’ narrative recall. Cognitive Development, 40, 68–81. https://doi.org/10.1016/j.cogdev.2016.08.005
Mayer, R. E. (2002). Multimedia learning. Psychology of learning and motivation (Vol. 41, pp. 85–139). Elsevier.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. The University of Chicago Press.
McNeil, N. M., Alibali, M. W., & Evans, J. L. (2000). The role of gesture in children’s comprehension of spoken language: Now they need it, now they don’t. Journal of Nonverbal Behavior, 24(2), 131–150. https://doi.org/10.1023/A:1006657929803
Paivio, A. (1990). Mental representations: A dual coding approach. Oxford University Press.
Pozzer-Ardenghi, L., & Roth, W.-M. (2004). Photographs in lectures: Gestures as meaning-making resources. Linguistics and Education, 15(3), 275–293. https://doi.org/10.1016/j.linged.2005.01.001
Ping, R. M., Goldin-Meadow, S., & Beilock, S. L. (2014). Understanding gesture: Is the listener’s motor system involved? Journal of Experimental Psychology: General, 143(1), 195–204. https://doi.org/10.1037/a0032246
Quandt, L. C., Marshall, P. J., Shipley, T. F., Beilock, S. L., & Goldin-Meadow, S. (2012). Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: An EEG study of action production and gesture observation. Neuropsychologia, 50, 2745–2751. https://doi.org/10.1016/j.neuropsychologia.2012.08.005
Sweller, N., Shinooka-Phelan, A., & Austin, E. (2020). The effects of observing and producing gestures on Japanese word learning. Acta Psychologica. https://doi.org/10.1016/j.actpsy.2020.103079
Tian, Y., & Bourguet, M. L. (2016). Lecturers’ hand gestures as clues to detect pedagogical significance in video lectures. ACM International Conference Proceeding Series. https://doi.org/10.1145/29709302970933
Valenzeno, L., Alibali, M. W., & Klatzky, R. (2003). Teachers’ gestures facilitate students’ learning: A lesson in symmetry. Contemporary Educational Psychology, 28(2), 187–204. https://doi.org/10.1016/S0361-476X(02)00007-3
Wakefield, E., Novack, M. A., Congdon, E. L., Franconeri, S., & Goldin-Meadow, S. (2018). Gesture helps learners learn, but not merely by guiding their visual attention. Developmental Science, 21(6), e12664. https://doi.org/10.1111/desc.12664
Willems, R. M., Özyürek, A., & Hagoort, P. (2007). When language meets action: The neural integration of gesture and speech. Cerebral Cortex, 17(10), 2322–2333. https://doi.org/10.1093/cercor/bhl141
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation and data collection were performed by Megan Phillips and all authors contributed to the data analysis process. The first draft of the manuscript was written by Nicole Dargue and Naomi Sweller and all authors commented on subsequent versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This study was approved by the appropriate institutional and/or national research ethics committee (Macquarie University Human Research Ethics Committee; Reference Number: 5201800134) and was performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or a comparable ethical standard.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A
Narrative gesture and non-gesture phrases (missing verbal information conditions)
Donald Duck had a garden full of watermelons and one of them had won a prize [first prize gesture] at a local fair. While watering his garden, Donald Duck noticed that he had run out of water, so he [skipped] over to a water station to refill his bucket.
When Donald Duck first got [pumped gesture] water out using the lever, the water did not fill up the bucket because the water only [dribbled out]. When Donald Duck tried to use the lever again, the water came out too far. This [frustrated] Donald because he could not seem to fill up his bucket with water.
After using the lever again, the water finally went into the bucket. However, the water kept [spurting out of the water station] and pushed the bucket over to a ledge, where it began to move [tilt back and forth gesture]. Donald rushed over to the ledge to try and stop the bucket from falling, but it was too late. The bucket had fallen off the ledge, spilling all of the water onto the ground.
Donald Duck [picked up gesture] took the bucket back to the water station. This time, Donald Duck moved the lever up and down so fast that the water station began to change shape [expand gesture], but only a drop of water came out. Donald duck looked into the tap to see why the water was not coming out, and all of a sudden, water began spurting out [gesture towards eye].
Donald Duck then moved the lever up and down as fast as it could go. A stream of water spurted out, and he chased it with his bucket [back and forth gesture]. He continued to chase the water all the way back to the water station, but he ran so fast that his beak got stuck in the nozzle and water burst out of him [gesture burst out of his nose]. Donald Duck slammed the bucket down in front of the water station and moved the lever up and down once more. The pressure in the water station built up so much that water came blasting out which caused Donald Duck to get pushed [gesture up in the sky].
When Donald Duck landed back on the ground, he noticed that his bucket had been filled up with water. As he went to get it, something happened to the bucket [the bucket broke gesture], leaving the water sitting there in the shape of a bucket. Donald Duck watched as the water burst. He looked down at the puddle quacking angrily and watched as [the water slowly disappeared into a hole in the ground].
Bold = Gesture Phrase
Italic = Non-Gesture Phrase
Appendix B
See Table 4
Rights and permissions
About this article
Cite this article
Dargue, N., Phillips, M. & Sweller, N. Filling in the gaps: observing gestures conveying additional information can compensate for missing verbal content. Instr Sci 49, 637–659 (2021). https://doi.org/10.1007/s11251-021-09549-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11251-021-09549-2