The effects of observing and producing gestures on Japanese word learning

Highlights

• Observing iconic gestures at encoding enhanced Japanese word learning.

• Reproducing an instructor's gestures at encoding enhanced Japanese word learning.

• Observing and reproducing gestures were equally beneficial across time points.

• Reproducing gestures increased spontaneous iconic gesture production at recall.

• Gesture production at recall mediated effects of encoding condition on recall.

Abstract

While observing and producing gesture can enhance native language learning, research in foreign word learning is limited. English-speaking university students were verbally presented with Japanese verbs with their English translations in one of three conditions: no gesture, observing an instructor's iconic gestures, or observing and reproducing the instructor's gestures. Participants verbally recalled the words immediately following training and after a one-week delay. Both observing only as well as observing and reproducing gestures at encoding enhanced learning and were equally beneficial for both the short and longer term beyond verbal learning alone. Reproducing the instructor's gestures during learning increased the rate of spontaneous iconic gesture production during recall, though the effects of this spontaneous gesture production at test were inconsistent. Findings emphasise the importance of an instructor's gesture production during foreign language learning, and suggest that motor imagery plays a key role in the beneficial effects of gesture on learning.

Introduction

Communication occurs through both verbal and non-verbal modalities, such as gesture. Across ages and cultures, people gesture to further illustrate their verbal message (Hostetter, 2011; Macedonia & von Kriegstein, 2012). Gestures are used not only when communicating in one's native language, but also in a foreign or second language (L2), frequently to compensate for speech difficulties by visually representing the meaning of words (Macedonia & Klimesch, 2014).

Gestures can benefit learners across a variety of tasks (see Dargue, Sweller, & Jones, 2019 for a review), including in educational settings such as when learning a second language (Gullberg, 1998, Gullberg, 2006, Gullberg, 2014; Gullberg, deBot, & Volterra, 2008; McCafferty, 2002). Observing and producing gestures both promote learning beyond verbal modality learning alone (e.g., Baills, Suárez-Gonzáles, Gonzáles-Fuente, & Prieto, 2018). However, research examining the role of gesture in L2 learning is limited (Macedonia & von Kriegstein, 2012). It is not yet clear whether observing or producing gestures is more effective as a language teaching tool (though see Morett, 2018), and whether these effects vary with time to recall. Further, examining gesture production at test, and how such gesture production may mediate the relationship between gesture observation/production during learning and recall at test is yet to be examined. This study examined the effects of observing and reproducing gestures during foreign word learning, whether such effects vary with time, and the potential mediation of observing or producing gestures on word learning via gesture production at test.

Vocabulary is important for effectively communicating meaning in any language. One of the difficulties that foreign language learners often face is the acquisition of novel vocabulary terms (Kelly, McDevitt, & Esch, 2009). Despite learners' efforts to acquire these words, memory for foreign language words often decays shortly after learning (Macedonia, Müller, & Friederici, 2011). Given the beneficial effects noted above of gestures on a variety of learning tasks, it is possible that encoding and retaining novel vocabulary terms may be facilitated by the presence of gesture during learning.

Examination of the effect of gesture in learning foreign words or sentences has been limited. Notably, the effect of gesture on L2 learning is often assessed in written formats (e.g. Kelly et al., 2009). Examinations of written recall only are restricted in their generalisability however, as language is used in both written and verbal formats. The extent to which gestures can benefit verbal recall remains unclear.

Recent research has examined novel L2 word learning in a dialogic task (Morett, 2018). Morett (2018) found that producing, but not viewing gestures affected communication and learning, and concluded the effects of producing gestures were stronger than the effects of observing gestures. However, the study only examined the effects of producing gestures while explaining words to another learner, and did not examine the effects of producing gestures while participants learned the words themselves. Furthermore, participants were not instructed to produce any specific type of gesture. Not all gestures are equivalent, and a consideration of gesture types is warranted.

Gestures can be classified into many non-mutually exclusive formats. Iconic gestures represent a concrete concept, visually recreating an aspect of the referent such as shape, size and movement (McNeill, 1992). For instance, a hand gesture which holds an imaginary glass and moves it towards the mouth can represent an action to drink. Given their link to concrete imagery, iconic gestures are those which may best facilitate the link between novel words and meanings. Such gestures are the focus of the current study. Other gesture types include metaphoric gestures (which represent abstract concepts), beat gestures (rhythmic hand movements) and deictic (pointing) gestures (McNeill, 1992).

Gestures can either be produced by others and observed by listeners, or produced by speakers themselves. The effect of both types of gesture on foreign word learning has been examined (e.g., Allen, 1995; De Nooijer, Van Gog, Paas, & Zwaan, 2013; Rowe, Silverman, & Mullan, 2013; Tellier, 2008), but with no attention to the potential mediating effect of producing gestures at test on the relationship between gesture observation and learning at training, and performance at test. We turn now to a discussion of observing and producing gestures on language learning, and proposed mechanisms underlying each function.

Observing gestures can benefit foreign word learning, tone identification and pronunciation (Allen, 1995; Gluhareva & Prieto, 2016). For example, observing syllabic beat gestures accompanying foreign words improves tone identification and word pronunciation (Baills et al., 2018). Mapping words to meaning is an important part of foreign language vocabulary acquisition (Jiang, 2002). The semantic representations that iconic gestures present can enable learners to build and access connections between words and meanings in their native language, making the memory of words more enduring (Kelly et al., 2009).

Foreign language learners benefit from observing iconic gestures, which can physically provide semantic representations of novel words (see Huang, Kim, & Christianson, 2018; Kelly et al., 2009). Establishing the connection between novel words and their meaning is a necessary process to integrate foreign language vocabulary (Allen, 1995). As such, observing iconic gestures can aid learners to internalise foreign words.

There are a number of methods through which observing gesture may enhance foreign language learning. Speech and gesture may work together to convey information through differing modalities. Dual coding theory and multimedia learning theory argue that listeners construct separate mental representations of speech and gesture content (Mayer, 2009; Paivio, 1990). Referential connections integrate the two mental representations, allowing the verbal and nonverbal systems to trigger activity in each other. If one mental representation is degraded or forgotten, the other may still be accessible to memory (Paivio, 1990).

It is also possible that observing gestures may elicit gesture production by learners, in turn enhancing learning. Observing a teacher's gestures can benefit learners by increasing the likelihood of their own spontaneous gesture production (Cook & Goldin-Meadow, 2006). That is, observing a teacher's gestures may play two important roles: not only through enhancing learners' comprehension of the spoken information, but also through encouraging learners to spontaneously produce gestures themselves. Producing gestures can itself lead to internalisation and enhanced retention of spoken information.

Producing or imitating gestures during learning can enhance word learning (Baills et al., 2018; Macedonia, Bergmann, & Roithmayr, 2014; Macedonia & Klimesch, 2014; Macedonia & Knösche, 2011). For example, words or sentences of an artificial language that participants learned by reproducing an instructor's gestures were recalled better over both the short-term for sentences (six days) (Macedonia & Knösche, 2011) and the long-term for words (444 days) (Macedonia & Klimesch, 2014) than those taught verbally. Participants in the 2011 study received daily training sessions, however the effect of producing gestures over verbal only learning did not appear until the third day of learning (Macedonia & Knösche, 2011), indicating that the effect of producing gestures may not emerge in the short-term.

Producing gestures has similarly found to benefit learning of non-artificial languages, such as Chinese tones (Baills et al., 2018). Furthermore, Hirata, Kelly, Huang, and Manansala (2014) explored whether observing and/or producing similar types of beat gestures enhanced learning of Japanese length contrasts. Observing and/or producing gestures improved vowel length identification, although it should be noted that the Hirata et al. (2014) study did not include a condition where participants neither observed nor produced gestures, meaning conclusions regarding the efficacy of gestures on foreign word learning remain tentative. Similarly, Kelly, Hirata, Manansala, and Huang (2014) examined the effects of observing and producing gestures on learning Japanese moras and syllables (but not whole words) and found similar results for both observing only and observing and producing gestures. Again however, this study only compared both observing and producing gestures with observing gestures only, without any no-gesture condition.

There are a number of mechanisms through which the production of gesture may enhance foreign language learning. Producing gestures may increase available cognitive resources (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001; Wilson, 2002). According to theories of embodied cognition, physical movements, including gesture, play an important role in shaping cognitive processes, and the body can be used to offload cognitive work onto the environment (Wilson, 2002). In this way, it may be possible to extend our limited cognitive capacity by externalising or embodying mental representations through gesture production (Wilson, 2002). If producing gesture lightens the load on cognitive resources, more resources can be allocated to the task, potentially improving memory performance (Goldin-Meadow, 2010).

It is also possible that producing gestures may facilitate access to stored information by providing a stable, external physical memory cue (Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014). Alibali, Kita, and Young (2000) argued that physically producing what is in one's mind through gestures enhances speech production by conceptualising thoughts. Gestures can perhaps alter the way we conceptualise information through activating, manipulating, packaging and exploring information (Kita, Alibali, & Chu, 2017). Given their link to concrete concepts, iconic gestures may be particularly well placed to provide such external memory cues. Iconic gestures can enhance word retrieval by cross-modally activating a word's concept, due to their physical representations of the semantic features of words (Rauscher, Krauss, & Chen, 1996). It is clear that observing and producing gestures both have the potential to enhance novel word learning. What is less clear however is whether producing gestures during learning has any beneficial effects beyond those of simply observing gestures, or whether producing gestures at test can mediate the relationship between observing or producing gestures at training and performance at test.

It has been proposed that there are perceptual, attentional, linguistic, spatial, memory and embodied/situated mechanisms underlying the effects of observing and producing gesture on learning (see Cook, 2018 for a review). Although the current paper cannot and does not attempt to distinguish between these mechanisms, motor imagery (see Section 1.3.1 above) and physical movement (see Section 1.4.1) may play key roles. A comparison of observing only (motor imagery) with both observing and producing gestures (motor imagery plus physical movement) will help to distinguish these mechanisms. Whether observing and producing gestures perform different functions and therefore have differing, or perhaps additive, effects, or they perform similar functions and therefore have similar effects, is not yet clear.

Faster processing time and enhanced memory performance have been found in action sentence recall in a native language (e.g., “lift the pencil”) when participants produced the corresponding action than when they learned the sentences verbally (Cohen, 1981; Masumoto et al., 2006; Mohr, Engelkamp, & Zimmer, 1989). Action and gesture both involve physical movement, and iconic gestures can have a clear resemblance to actions. Producing actions and iconic gestures may, therefore, be similarly effective (Macedonia & Knösche, 2011).

Indeed, some studies have found beneficial effects of producing actions on recall of action sentences beyond effects of only observing an experimenter perform the actions (Engelkamp & Zimmer, 1997; Mulligan & Hornstein, 2003). Similarly, the work of Morett (2018) suggested that the effects of producing gestures were stronger than those of observing gestures in L2 word learning (see Section 1.1 above). Conversely, other studies have found no beneficial effects of producing gestures beyond observing or imagining gestures (e.g., Brooks, Barner, Frank, & Goldin-Meadow, 2018; Cohen, 1981; Kamermans et al., 2019; Kormi-Nouri, 2000). Kormi-Nouri (2000) proposed that physical movement is not crucial, but rather that motor imagery, a mental simulation of performing the action, is sufficient to promote recall.

The Gesture as Simulated Action (GSA) framework holds that both observing and producing gesture elicits motor imagery (Hostetter & Alibali, 2008; see also Wu & Coulson, 2014, for a discussion of iconic gestures promoting image-based simulations). Motor imagery, therefore, occurs regardless of whether or not gestures are overtly produced, although the GSA framework does not preclude an additional beneficial effect of producing gestures beyond that of motor imagery alone. In sum, it is unclear whether it is motor imagery rather than physical movement that plays a role in the positive effect of gesture production beyond verbal only learning (Kormi-Nouri, 2000), or whether physical movement through producing gestures could enhance Japanese word learning beyond only observing gestures (Engelkamp & Zimmer, 1997).

The present study examined the effects of observing and reproducing iconic gestures on learning Japanese verbs. It examined whether learning words with accompanying iconic gestures at encoding promotes learning beyond hearing the words without gestures, as well as whether there is a larger effect of reproducing an instructor's gestures, than of only observing the gestures, both in the short- and longer-term. As past studies indicate that the beneficial effect of gesture production takes time to emerge, we used a one week gap between two memory tests.

The study also examined whether observing or reproducing gestures at encoding affects learners' spontaneous gesture production during retrieval, and whether rates of spontaneous gesture production change with time. Iconic gestures are of particular interest given their conceptual relationship with concrete concepts such as the verbs presented in the current study. Finally, we examined the relationship between gesture observation, spontaneous gesture production at retrieval and recall.

It was expected that: 1) observing or reproducing iconic gestures at encoding would enhance word recall to a greater extent than speech only learning; 2) word recall would be greater for participants who reproduced iconic gestures at encoding than those who only observed gestures; 3) while recall would decrease with time, this decrease would be smaller for participants who observed or reproduced iconic gestures at encoding than those who received speech only; 4) spontaneous gesture production at retrieval would be greater for participants who observed or reproduced the instructor's gestures at encoding than those who received speech only and 5) a positive association between the number of spontaneous gestures produced at retrieval and words recalled would be found, as well as a positive mediating effect of gestures produced at test on the relationship between gesture condition at encoding and recall.