Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter Mouton June 14, 2021

Distant time, distant gesture: speech and gesture correlate to express temporal distance

  • Daniel Alcaraz Carrión ORCID logo EMAIL logo and Javier Valenzuela
From the journal Semiotica

Abstract

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance.


Corresponding author: Daniel Alcaraz Carrión, University of Wisconsin-Madison, Wisconsin, USA, E-mail:

  1. Research funding: This work was supported by the Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación & FEDER/UE funds (grant number PGC2018-1551 097658- B-100).

References

Beattie, Geoffrey & Heather Shovelton. 2011. An exploration of the other side of semantic communication: How the spontaneous movements of the human hand add crucial meaning to narrative. Semiotica 184. 33–51.10.1515/semi.2011.021Search in Google Scholar

Beattie, Geoffrey. 2016. Rethinking body language. London: Routledge.10.4324/9781315880181Search in Google Scholar

Bender, Andrea & Sieghard Beller. 2014. Mapping spatial frames of reference onto time: A review of theoretical accounts and empirical findings. Cognition 132(3). 342–382. https://doi.org/10.1016/j.cognition.2014.03.016.Search in Google Scholar

Boroditsky, Lera. 2000. Metaphoric structuring: Understanding time through spatial metaphors. Cognition 75. 1–28. https://doi.org/10.1016/s0010-0277(99)00073-6.Search in Google Scholar

Boroditsky, Lera. 2001. Does language shape thought?: English and Mandarin speaker’s conceptions of time. Cognitive Psychology 43. 1–22. https://doi.org/10.1006/cogp.2001.0748.Search in Google Scholar

Boroditsky, Lera & Michael Ramscar. 2002. The roles of body and mind in abstract thought. Psychological Science 13(2). 185–189. https://doi.org/10.1111/1467-9280.00434.Search in Google Scholar

Bressem, Jana & Silva Ladewig. 2011. Rethinking gesture phases: Articulatory features of gestural movement? Semiotica 184(1/4). 53–91.10.1515/semi.2011.022Search in Google Scholar

Bressem, Jana, Silva Ladewig & Cornelia Müller. 2013. Linguistic annotation system for gestures (LASG). In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill & Jana Bressem (eds.), Body – language – communication: An international handbook on multimodality in human interaction, 2nd ed. vol. 38, 1098–1124. Berlin: De Gruyter Mouton.10.1515/9783110261318.1098Search in Google Scholar

Bylund, Emanuel, Pascal Gygax, Steven Samuel & Panos Athanasopoulos. 2019. Back to the future? The role of temporal focus for mapping time onto space. Quarterly Journal of Experimental Psychology 73(2). 174–182.10.1177/1747021819867624Search in Google Scholar

Cai, Zhenguang G., Louise Connell & Judith Holler. 2013. Time does not flow without language: Spatial distance affects temporal duration regardless of movement or direction. Psychonomic Bulletin & Review 20. 973–980. https://doi.org/10.3758/s13423-013-0414-3.Search in Google Scholar

Calbris, Geneviève. 2008. From left to right … Coverbal gestures and their symbolic use of space. In Cornelia Müller & Alan Cienki (eds.), Metaphor and gesture, 27–55. Amsterdam & Philadelphia: John Benjamin.10.1075/gs.3.05calSearch in Google Scholar

Casasanto, Daniel & Lera Boroditsky. 2008. Time in the mind: Using space to think about time. Cognition 106. 579–593. https://doi.org/10.1016/j.cognition.2007.03.004.Search in Google Scholar

Casasanto, Daniel & Kyle Jasmin. 2012. The hands of time: Temporal gestures in English speakers. Cognitive Linguistics 23(4). 643–674. https://doi.org/10.1515/cog-2012-0020.Search in Google Scholar

Cienki, Alan & Cornelia Müller. 2008. Metaphor and gesture. Amsterdam, Philadelphia: John Benjamins Pub. Co.10.1075/gs.3Search in Google Scholar

Cienki, Alan, Aliyah Morgenstern, Cornelia Müller & Dominique Boutet. 2016. Linguistic aspect, tense and gestural movement quality in French, German, and Russian utterances, vol. 7. Paris: International Society of Gestures Studies.Search in Google Scholar

Cienki, Alan. 1998. Metaphoric gestures and some of their relations to verbal metaphoric expressions. In Jean-Pierre Koenig (ed.), Discourse and cognition: Bridging the gap, 189–204. Stanford, CA: CSLI Publications.Search in Google Scholar

Cienki, Alan. 2008. Why study metaphor and gesture? In Alan Cienki & Cornelia Müller (eds.), Metaphor and gesture, 5–25. Cambridge: Cambridge University Press. https://doi.org/10.1075/gs.3.04cie.Search in Google Scholar

Cienki, Alan. 2013. Cognitive linguistics: Language and gestures as expressions of conceptualisation. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill & Sedinha Tessendorf (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 182–201. Berlin: De Gruyter Mouton.Search in Google Scholar

Cooperrider, Kensy & Rafael Núñez. 2009. Across time, across the body: Transversal temporal gestures. Gesture 9(2). 181–206. https://doi.org/10.1075/gest.9.2.02coo.Search in Google Scholar

Cohen, Doron, Geoffrey Beattie & Heather Shovelton. 2010. Nonverbal indicators of deception: How iconic gestures reveal thoughts that cannot be suppressed. Semiotica 182. 133–174.10.1515/semi.2010.055Search in Google Scholar

Colapietro, Vincent. 2001. A lantern for the feet of inquirers: The heuristic function of the Peircean categories. Semiotica 136(1/4). 201–216.10.1515/semi.2001.074Search in Google Scholar

Coventry, Kenny R., Berenice Valdés, Alejandro Castillo & Pedro Guijarro-Fuentes. 2008. Language within your reach: Near–far perceptual space and spatial demonstratives. Cognition 108. 889–895. https://doi.org/10.1016/j.cognition.2008.06.010.Search in Google Scholar

Davis, Stephen Boyd. 2012. History on the line: Time as dimension. Design Issues 26(4). 4–17. https://doi.org/10.1162/desi_a_00171.Search in Google Scholar

Danaher, David. 1998. Peirce’s semiotic and cognitive metaphor theory. Semiotica 119(1/2). 171–207.Search in Google Scholar

de la Fuente, Juanma, Julio Santiago, Antonio Román, Cristina Dumitrache & Daniel Casasanto. 2014. When you think about it, your past is in front of you: How culture shapes spatial conceptions of time. Psychological Science 29. 1682–1690. https://doi.org/10.1177/0956797614534695.Search in Google Scholar

Diessel, Holger. 1999. Demonstratives: Form, function, and grammaticalization. Amsterdam/Philadelphia: John Benjamins.10.1075/tsl.42Search in Google Scholar

Diessel, Holger. 2006. Demonstratives, joint attention, and the emergence of grammar. Cognitive Linguistics 17(4). 463–489. https://doi.org/10.1515/cog.2006.015.Search in Google Scholar

Diessel, Holger. 2013. Distance contrasts in demonstratives. In Matthew S. Dryer & Martin Haspelmath (eds.), The world atlas of language structures online. Leipzig: Max Planck Institute for Evolutionary Anthropology.Search in Google Scholar

Duffy, Sarah. 2014. The role of cultural artifacts in the interpretation of metaphorical expressions about time. Metaphor and Symbol 29(2). 94–112. https://doi.org/10.1080/10926488.2014.889989.Search in Google Scholar

Farias, Priscila & Joao Queiroz. 2006. Images, diagrams, and metaphors: Hypoicons in the context of Peirce’s sixty-six-fold classification of signs. Semiotica 162(1/4). 287–307.10.1515/SEM.2006.081Search in Google Scholar

Fauconnier, Gilles & Mark Turner. 2008. Rethinking metaphor. New York: Cambridge University Press.10.1017/CBO9780511816802.005Search in Google Scholar

Goldin-Meadow, Susan. 2003. Hearing gesture: How our hands help us think. Cambridge: Belknap Press of Harvard University Press.10.1037/e413812005-377Search in Google Scholar

Goldin-Meadow, Susan. 2004. Gestures role in the learning process. Theory Into Practice 43. 314–321. https://doi.org/10.1207/s15430421tip4304_10.Search in Google Scholar

Gullberg, Marianne & Sotaro Kita. 2009. Attention to speech-accompanying gestures: Eye movements and information uptake. Journal of Nonverbal Behavior 33. 251–277. https://doi.org/10.1007/s10919-009-0073-2.Search in Google Scholar

Gullberg, Marianne & Kenneth Holmqvist. 1999. Keeping an eye on gestures: Visual perception of gestures in face-to-face communication. Pragmatics and Cognition 7. 35–63. https://doi.org/10.1075/pc.7.1.04gul.Search in Google Scholar

Gullberg, Marianne & Kenneth Holmqvist. 2002. Visual attention towards gestures in face-to-face interaction vs. on screen. In Ipke Wachsmuth & Timo Sowa (eds.), Gesture and sign language in human-computer interaction. GW 2001. Berlin, Heidelberg: Springer.10.1007/3-540-47873-6_23Search in Google Scholar

Gullberg, Marianne & Kenneth Holmqvist. 2006. What speakers do and what listeners look at: Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition 14. 53–82. https://doi.org/10.1075/pc.14.1.05gul.Search in Google Scholar

Iriskhanova, Olga & Alan Cienki. 2018. The semiotics of gestures in cognitive linguistics: Contribution and challenges. Voprosy Kognitivnoy Lingvistiki 4. 25–36. https://doi.org/10.20916/1812-3228-2018-4-25-36.Search in Google Scholar

Iverson, Jana & Susan Goldin-Meadow. 1998. Why people gesture as they speak. Nature 396. 228. https://doi.org/10.1038/24300.Search in Google Scholar

Kemmerer, David. 1999. “Near” and “far” in language and perception. Cognition 73(1). 35–63. https://doi.org/10.1016/s0010-0277(99)00040-2.Search in Google Scholar

Kendon, Adam. 1980. Gesture and speech: Two aspects of the process of utterance. In Mary R. Key (ed.), Nonverbal communication and language, 207–227. The Hague: Mouton.10.1515/9783110813098.207Search in Google Scholar

Kendon, Adam. 2000. Language and gesture: Unity or duality. In David McNeill (ed.), Language and gesture: Window into thought and action, 47–63. Cambridge: Cambridge University Press.10.1017/CBO9780511620850.004Search in Google Scholar

Kendon, Adam. 2004. Gesture: Visible action as utterance. Cambridge: Cambridge University Press.10.1017/CBO9780511807572Search in Google Scholar

Kranjec, Alexander & Anjan Chatterjee. 2010. Are temporal concepts embodied? A challenge for cognitive neuroscience. Frontiers in Psychology 1. 240. https://doi.org/10.3389/fpsyg.2010.00240.Search in Google Scholar

Kranjec, Alexander & Laraine McDonough. 2011. The implicit and explicit embodiment of time. Journal of Pragmatics 43(3). 735–748. https://doi.org/10.1016/j.pragma.2010.07.004.Search in Google Scholar

Lakoff, George & Mark Johson. 1980. Metaphors we live by. Chicago: Chicago University Press.Search in Google Scholar

Lakoff, George. 1987. Women, fire, and dangerous things: what categories reveal about the mind. Chicago: University of Chicago Press.10.7208/chicago/9780226471013.001.0001Search in Google Scholar

Lapaire, Jean-Remi. 2016. From ontological metaphor to semiotic make-believe: Giving shape and substance to fictive objects of conception with the globe gesture. Signo 41. 70. https://doi.org/10.17058/signo.v41i70.6413.Search in Google Scholar

Levinson, Stephen & Asifa Majid. 2013. The island of time: Yélî Dnye, the language of Rossel island. Frontiers in Psychology 4. 61. https://doi.org/10.3389/fpsyg.2013.00061.Search in Google Scholar

Matlock, Teenie, Michael Ramscar & Lera Boroditsky. 2005. On the experiential link between spatial and temporal language. Cognitive Science 29(4). 655–664. https://doi.org/10.1207/s15516709cog0000_17.Search in Google Scholar

McNeill, David. 1985. So you think gestures are nonverbal? Psychological Review 92. 350–371. https://doi.org/10.1037/0033-295x.92.3.350.Search in Google Scholar

McNeill, David. 1992. Hand and mind. Chicago: University of Chicago Press.Search in Google Scholar

McNeill, David. 2005. Gesture and thought. Chicago: University of Chicago Press.10.7208/chicago/9780226514642.001.0001Search in Google Scholar

Mittelberg, Irene. 2019. Peirce’s universal categories: On their potential for gesture theory and multimodal analysis. Semiotica 228. 193–222. https://doi.org/10.1515/sem-2018-0090.Search in Google Scholar

Mittelberg, Irene. 2008. Peircean semiotics meets conceptual metaphor: Iconic modes in gestural representations of grammar. In Alan Cienki & Cornelia Müller (eds.), Metaphor and gesture, 115–154. Amsterdam & Philadelphia: John Benjamins.10.1075/gs.3.08mitSearch in Google Scholar

Moore, Kevin. 2006. Space-to-time mappings and temporal concepts. Cognitive Linguistics 17. 199–244. https://doi.org/10.1515/cog.2006.005.Search in Google Scholar

Moore, Kevin. 2011. Ego-Perspective and field-based frames of reference: Temporal meanings of FRONT in Japanese, Wolof and Aymara. Journal of Pragmatics 43(3). 759–776. https://doi.org/10.1016/j.pragma.2010.07.003.Search in Google Scholar

Ng, Melvin R., Winston D. Goh, Melvin J. Yap, Chi-Sing Tse & Wing-Chee So. 2017. We think about temporal words: A gestural priming study in English and Chinese. Friontiers in Psychology 8. 974. https://doi.org/10.3389/fpsyg.2017.00974.Search in Google Scholar

Nöth, Winfried. 1990. Handbook of semiotics. Bloomington: Indiana University Press.10.2307/j.ctv14npk46Search in Google Scholar

Nöth, Winfried. 1999. Peircean semiotics in the study of iconicity in language. Transactions of the Charles S. Peirce Society 35(3). 613–619.Search in Google Scholar

Núñez, Rafael & Eve Sweetser. 2006. With the future behind them: Converge evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time. Cognitive Science 30. 1–49.10.1207/s15516709cog0000_62Search in Google Scholar

Özyrükek, Asli. 2014. Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour. Philosophical Transactions of the Royal Society of London Series B Biological Sciences 369. 1651.10.1098/rstb.2013.0296Search in Google Scholar

Pagán Cánovas, Cristóbal, Javier Valenzuela, Alcaraz Carrión Daniel, Olza Inés & Ramscar Michael. 2020. Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions. PloS One 15(6). e0233892. https://doi.org/10.1371/journal.pone.0233892.Search in Google Scholar

Peirce, Charles S. 1931–1966. The collected papers of Charles S. Peirce. In Charles Hartshorne, Paul Weiss & Arthur W. Burks (eds.), vol. 8. Cambridge: Harvard University Press [Reference to Peirce’s papers will be designated CP followed by volume and paragraph number].Search in Google Scholar

Santiago, Julio, Juan Lupiáñez, Elvira Pérez & Maria José Funes. 2007. Time also flies from left to right. Psychonomic Bulletin & Review 14(3). 512–516. https://doi.org/10.3758/bf03194099.Search in Google Scholar

Torralbo, Ana, Julio Santiago & Juan Lupiañez. 2006. Flexible conceptual projection of time onto spatial frames of reference. Cognitive Science 30. 745–757. https://doi.org/10.1207/s15516709cog0000_67.Search in Google Scholar

Valenzuela, Javier, Cristobal Pagán-Cánovas, Inés Olza & Daniel Alcaraz. 2020. Gesturing in the wild: Spontaneous gestures co-occurring with temporal demarcative expressions provide evidence for a flexible mental timeline. Review on Cognitive Linguistics 18(2). 289–316. https://doi.org/10.1075/rcl.00061.val.Search in Google Scholar

Walker, Esther & Kensy Cooperrider. 2016. The continuity of metaphor: Evidence from temporal gestures. Cognitive Science 40(2). 481–495. https://doi.org/10.1111/cogs.12254.Search in Google Scholar

Received: 2019-11-22
Accepted: 2020-08-17
Published Online: 2021-06-14
Published in Print: 2021-07-27

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 29.3.2024 from https://www.degruyter.com/document/doi/10.1515/sem-2019-0120/html
Scroll to top button