Activating words beyond the unfolding sentence: Contributions of event simulation and word associations to discourse reading

Highlights

• We tested the contribution of associations to discourse reading using ERPs.

• Targets were preceded by associatively related words in coherent discourse context.

• Unexpected words elicited a reduced N400 if related to the described event.

• A similar pattern was observed when event simulation was restricted.

• Event simulation and associations likely both contribute to discourse reading.

Abstract

Previous studies have shown that during comprehension readers activate words beyond the unfolding sentence. An open question concerns the mechanisms underlying this behavior. One proposal is that readers mentally simulate the described event and activate related words that might be referred to as the discourse further unfolds. Another proposal is that activation between words spreads in an automatic, associative fashion. The empirical support for these proposals is mixed. Therefore, theoretical accounts differ with regard to how much weight they place on the contributions of these sources to sentence comprehension. In the present study, we attempted to assess the contributions of event simulation and lexical associations to discourse reading, using event-related brain potentials (ERPs). Participants read target words, which were preceded by associatively related words either appearing in a coherent discourse event (Experiment 1) or in sentences that did not form a coherent discourse event (Experiment 2). Contextually unexpected target words that were associatively related to the described events elicited a reduced N400 amplitude compared to contextually unexpected target words that were unrelated to the events (Experiment 1). In Experiment 2, a similar but reduced effect was observed. These findings support the notion that during discourse reading event simulation and simple word associations jointly contribute to language comprehension by activating words that are beyond contextually congruent sentence continuations.

Introduction

Written text comprehension is fast and efficient. Ziefle (1998) and Noyes and Garland (2008) estimated that the average adult reads prose at a speed of 250–300 words per minute (see Brysbaert, 2019, for a lower estimate). Why is it that we read text with such ease and at such a high speed? One reason, which has turned into a prominent feature of many models of sentence comprehension, is that we often pre-activate words beyond the unfolding sentence (e.g., Ferreira and Chantavarin, 2018; Pickering and Garrod, 2013; Pickering and Gambi, 2018; Schwanenflugel and Shoben, 1985; Van Berkum et al., 2005; Wicha et al., 2003, 2004).

Previous studies have led to a good understanding of the contents language users may pre-activate during comprehension (i.e. what is pre-activated, Arai and Keller, 2013; Chen et al., 2005; Federmeier et al., 2002; Laszlo et al., 2012; Rommers et al., 2013; Staub and Clifton, 2006) as well as the cues used to pre-activate linguistic or non-linguistic information (e.g., Federmeier and Kutas, 1999; Knoeferle et al., 2005; Van Berkum et al., 2005). However, the underlying mechanisms, i.e. the processes connecting cues and contents, still remain poorly understood. In particular, experimental studies providing evidence for specific mechanisms contributing to (predictive) language comprehension are sparse. The goal of the present study was to map out the contributions of two proposed mechanisms, event simulation and simple associations, to discourse reading.

Researchers have long been highlighting the importance of event knowledge—referring to knowledge about real world events and regularities, as well as objects, persons and locations involved in them—for language and cognitive processing more broadly (Barsalou, 2008; Kahneman and Tversky, 1973; Mandler, 1984; Minsky, 1974; Rumelhart, 1980; Schank and Abelson, 1977; Zacks and Tversky, 2001). Concerning the involvement of event knowledge in language processing, it has been proposed that language users engage in simulating real-world events and pre-activate linguistic representations that are likely to occur in that event (Elman, 2009; Elman and McRae, 2019; Huettig, 2015).

In an electroencephalography (EEG) study, Metusalem et al. (2012) reported experimental evidence demonstrating that event simulation contributes to activating referents that are likely to be mentioned in an unfolding discourse. They recorded the N400, which is a negative-going deflection in the EEG signal that peaks around 400 ms after stimulus onset. It is considered an index of semantic processing (Kutas and Hillyard, 1980; Kutas and Federmeier, 2011, for review). The N400 ERP component is typically distributed over centro-parietal electrodes. Participants in the study by Metusalem et al. read passages consisting of two introductory sentences and one target sentence. Taken together, the three sentences described typical events such as ‘The parents were very excited about their new baby girl. One of the first things they did was to get her baptized in their church. The baby liked baths, so she smiled when she was sprinkled with water/priest/dentist on her forehead’. The authors observed a three-way split in N400 amplitude: Expected targets (‘water’ in the example) elicited the smallest N400 amplitude. Contextually unexpected words (‘dentist’) elicited the largest N400 amplitude. Interestingly, contextually unexpected targets (‘priest’), which were related to the described event, elicited an attenuated N400 amplitude, which lay in between these two extremes. In their second experiment, participants read the target sentences presented in isolation (without the first two introductory sentences). Now the N400 elicited by the unexpected but event-related word (‘priest’) condition did not differ from the N400 elicited by the unexpected event-unrelated word (‘dentist’). Metusalem et al. concluded that readers used generalized knowledge to simulate the described event, thereby constraining the activation of potentially upcoming referents, i.e. event-related words beyond the most expected continuation of the target sentences. Specifically, they reasoned that the N400 reduction "[ …] suggests that event knowledge also is an important knowledge source for driving linguistic prediction" (p. 559).

Using latent semantic analysis norms (LSA, Landauer and Dumais, 1997), Metusalem et al. conducted a post-hoc analysis assessing the potential influence of word associations on the observed pattern of results. To that end, they calculated the cosine distance between the target words and the entire preceding context. Event-related targets were significantly more strongly associated with the preceding words than the completely unrelated targets. In a next step, the authors removed twelve items, which showed the largest differences between LSA association scores for the event-related and event-unrelated targets, and re-ran their ERP analysis. The analysis revealed the same three-way split in N400 amplitude as previously, i.e. a significant difference between event-related and event-unrelated unexpected target words. Metusalem and colleagues thus concluded that the effect seen in the event-related condition was unlikely to be due to associative priming, i.e. simple word associations between words in the preceding context and the critical targets.

Importantly, while event simulation is likely to be a strong contributor to language comprehension, to be able to exclude the contribution of associations to discourse reading, a more direct test than a post-hoc analysis is needed. Observing evidence for the involvement of associations in discourse reading would be in line with theoretical frameworks that assume that simple word associations play a role in anticipatory language comprehension, albeit as one of multiple contributing mechanisms (Huettig, 2015; Kuperberg, 2007; Pickering and Gambi, 2018; Pickering and Garrod, 2013; cf. Bar, 2007, 2009).

As described above, one way of operationalizing word associations, or the associative strength between two words, is by calculating their co-occurrence frequency based on large corpora of written texts (such as LSA, Landauer and Dumais, 1997). Another common way is to use experimental tasks such as free association (De Deyne et al., 2013), where participants are provided with a cue word and have to generate one or multiple associates that come to mind. The general assumption is that words that co-occur frequently have a strong associative link and that such links are utilized in various cognitive tasks. Note that even though the nature of such links is often semantic, an associative relationship between two words does not necessarily require a semantic relation. For example, the classic doctor-nurse pair may be associatively related because of a semantic relationship and lexical co-occurrence, whereas pairs such as ‘good morning’ are associatively related purely on the basis of lexical occurrence. The present study concerns relationships of the former type, and we do not differentiate between purely semantic and purely associative (without a semantic relationship) links.

Neely (1991) proposed that the activation of one word leads to the retrieval/activation of another word in an automatic or ‘expectancy-based’ fashion. In a similar vein, Pickering and Garrod (2013), Huettig (2015), Kuperberg (2007), Pickering and Gambi (2018) assume that associations contribute to language comprehension, and specifically anticipatory processing. For example, these accounts assume that ‘church’ and ‘priest’ (as in the stimulus example by Metusalem et al., 2012) are words that often occur together; they thus have a strong associative link. The proposal is that activation spreading via the associative link, upon encountering one word, contributes to the (partial) pre-activation of the other word, irrespective of the unfolding sentence.

Electrophysiological studies investigating associative priming in word pairs such as ‘church-priest’ have demonstrated that target words elicit a reduced N400 amplitude when preceded by associated primes as compared to unassociated primes (Bentin et al., 1985; Bentin, 1987; Rugg, 1987). For example, Van Petten (2014) showed that free association strength (e.g. Nelson et al., 2004), and corpus-based measures of association, such as LSA, accounted for substantial amounts of variance in the amplitude of the N400 component elicited by the second word in 303 word pairs. These and many related behavioral studies, employing different paradigms and techniques (see Hutchinson, 2003; Neely, 1991, for review), suggest that associative relationships facilitate word processing.

A recurring issue in the current literature concerns the contribution of simple associations to comprehension and prediction of linguistic stimuli more complex than word pairs (see Ledoux et al., 2006, for review). To investigate this issue, previous studies have typically manipulated the associative relationship in word pairs (associated vs. unassociated) and the words’ fit with the sentential context (congruent vs. incongruent). Pioneering work was carried out by Van Petten (1993), who compared the influence of associations with the influence of sentential contexts on the amplitude of the N400 component. An example of her stimuli is given in (1). Associated words are marked in italics.

• Associated/congruent: When the moon is full it is hard to see many stars or the Milky Way.

• Unassociated/congruent: When the insurance investigators found that he'd been drinking they refused to pay the claim.

• Associated/anomalous: When the moon is rusted it is available to buy many stars or the Santa Ana.

• Unassociated/anomalous: When the insurance supplies explained that he'd been complaining they refused to speak the keys.

Van Petten reported reductions in the N400 amplitude for the associated/congruent, the unassociated/congruent and the associated/anomalous conditions, relative to the unassociated/anomalous condition. She reasoned that the N400 reduction in the unassociated/congruent condition could be attributed to the sentential context and that the N400 reduction in the associated/anomalous condition was due to lexical association. Crucially, she found that the N400 amplitude reduction in the associated/congruent condition was larger than the reduction in the remaining two conditions, which she argued reflected the additive effects of sentential context and lexical association.

Coulson et al. (2005) further explored the electrophysiological signature of associative priming in processing associated word pairs in isolation and in simple sentence contexts. Employing a lateralized half field manipulation, they additionally investigated the engagement of the two brain hemispheres subserving associative priming and the integration of sentential constraints. Their participants read pairs of associated and unassociated words in isolation. The authors reported that the second word elicited more positive ERPs in associated than in unassociated pairs. In their second experiment, the same word pairs were embedded in simple sentences. Similar to Van Petten (1993), Coulson et al. crossed the experimental factors associative relationship and sentential fit. Their results showed that processing was primarily influenced by the words’ fit with the sentential contexts. However, the authors also reported subtle effects of associative priming as indexed by a more positive N400 amplitude for associated compared to unassociated pairs and a late positive component elicited by the associated but not by unassociated pairs. The influence of associative priming was particularly pronounced when the word pairs occurred in incongruent sentence contexts. With regard to hemispheric differences, Coulson and colleagues observed that after presentation to the right visual field, lexical associations had an effect in incongruous but not in congruous sentence completions. After left visual field presentation, lexical associations showed an effect in both congruous and incongruous sentences (see Beeman, 1993; Chiarello et al., 2001, for further discussion of hemispheric contributions to sentence processing).

Using visual world eye-tracking (Huettig et al., 2011, for review), Kukona et al. (2011) provided experimental evidence supporting the notion that associations and combinatorial event information jointly contribute to anticipatory processing during comprehension. The authors contrasted the influence of associative priming with the influence of event-based context on sentence comprehension. Their participants listened to sentences containing a critical verb (e.g., “arrest”) such as “Toby arrests the crook” while looking at visual scenes including verb-related agents and patients (e.g., a policeman and a crook). Kukona and colleagues observed anticipatory eye movements to both agents and patients although the agent role had already been filled (Toby). They concluded that anticipatory eye gaze was influenced by simple associative relationships between the words (e.g., arrest-policeman) and by event knowledge. Crucially, associative priming showed an effect even though it conflicted with the event built up by the sentential context.

Taken together, the studies by Van Petten, Coulson et al. and Kukona et al. showed that, at the sentence level, contextual constraints exert a substantial influence on language comprehension and language-mediated anticipatory eye movements. However, there is also some evidence for the modulating influence of simple word associations (but see Paczynski and Kuperberg, 2012). The data by Coulson et al. suggest that these appear to have an impact especially when the target words are incongruent with the sentence context they appear in.

Previous investigations have shown that discourse context beyond the level of single sentences impact the comprehension process as measured using the N400 component. Hagoort et al., (2004) asked Dutch participants to read sentences such as ‘Dutch trains are yellow/white/sour and very crowded’. The participants in that study knew that Dutch trains are typically yellow. Thus, the color word ‘white’ was a violation of their world knowledge and yielded an N400 similar to that elicited by the word ‘sour’. The authors argued that comprehenders immediately integrated word meanings and world knowledge. The same lab showed in a follow-up study that the critical N400 amplitude was attenuated when the target sentence was preceded by a mitigating context. Thus, reading ‘The coming world championships are one big national spectacle. The Dutch railways have painted the Dutch flag on their trains’ prior to the target sentence reduced participants' N400 amplitude in response to the word ‘white’ in the sentence, most likely because the Dutch flag includes the color white (Hald et al., 2007; see also Van Berkum et al., 1999). Note that in these and many similar studies the influence of the discourse context (e.g. simulation of an event) and the influence of associations between words in the discourse and the target words could not be separated. Previous attempts to do so yielded mixed results. There is consensus among researchers that simple associative relationships between words cannot fully account for the N400 amplitude modulations observed when critical target words are preceded by related discourse. However, whether or not associations contribute to discourse comprehension and if so, how strongly, is a matter of some debate.

To tease apart the effects of lexical association and the effects of discourse context, Camblin et al. (2007) orthogonally manipulated lexical associations between words and the words’ congruence with the event described in the discourse. In a series of experiments recording ERPs and eye movements during reading, they investigated how lexical-level effects interacted with the effects of discourse context. The authors embedded associated and unassociated word pairs in sentences that were coherent and locally congruent. The critical words were either congruous or incongruous with the discourse-level context (see (2) for an example).

• Associated/congruent: Lynn had gotten a sunburn at the beach. Nothing she tried would help her dry and irritated skin. Lynn couldn't stop scratching her arms and legs.

• Unassociated/congruent: Lynn had gotten a sunburn at the beach. Nothing she tried would help her dry and irritated skin. Lynn couldn't stop scratching her arms and nose.

• Associated/incongruent: Lynn's wool sweater was uncomfortable and itchy. She fidgeted as the rough material irritated her skin. Lynn couldn't stop scratching her arms and legs.

• Unassociated/incongruent: Lynn's wool sweater was uncomfortable and itchy. She fidgeted as the rough material irritated her skin. Lynn couldn't stop scratching her arms and nose.

Their analyses revealed independent effects of discourse congruence and lexical associations. Violations of discourse congruence had early and lingering effects on both ERP and eye-tracking measures, whereas the effects of associations were more fragile and particularly evident in scenarios in which the discourse context was not cohesive. Camblin et al. (2007; see Ledoux et al., 2006; Brothers et al., 2015; Otten and van Berkum, 2007, 2008, for similar conclusions) concluded that when a cohesive, congruent discourse model can be constructed, it may override associative facilitation, and effects of association “[do] not contribute to processing of words in sentences that are part of a larger discourse” (p. 126).

In a follow-up study, Boudewyn et al. (2011) used the same discourses as Camblin et al. in spoken form. In contrast to Camblin et al. (2007), they observed an interaction between lexical association and discourse congruency. Boudewyn and colleagues argued that local lexical associations and overall discourse congruence may each exert their own influence on incoming words during discourse comprehension and that these effects may be additive when the two sources of information are consistent with one another. In another study, Boudewyn and colleagues (Boudewyn et al., 2013) further demonstrated that the size of association effects during spoken discourse comprehension is subject to individual differences. Specifically, they showed that lexical associations had larger effects in individuals with lower working memory capacities as – arguably – these participants were less able to maintain discourse relationships during its unfolding. Processing in these participants was thus predominantly driven by relationships between individual words in the discourse (see also Sanford et al., 2011; Sanford and Garrod, 1998). Note, however, that the authors also observed a main effect of associations, irrespective of the individual variation between participants.

In sum, previous studies have clearly established that utilizing generalized knowledge during sentence and discourse comprehension, i.e. simulating the described event, exert powerful influences on language comprehension (McRae and Matsuki, 2009, for review). However, the experimental results with regard to the contribution of word associations to discourse comprehension are mixed: While some studies reported significant contributions in the spoken modality (Boudewyn et al., 2011, 2013), other studies in the visual domain ascribe only a subtle influence to lexical associations, which might be overridden by the effects of discourse context (Brothers et al., 2015; Camblin et al., 2007; Otten and van Berkum, 2007, 2008). Interestingly, one condition under which associative priming has repeatedly been shown to impact language comprehension is when the critical target words are incongruent with the local contexts they are embedded in (e.g., Camblin et al., 2007; Van Petten, 1993).

In light of this finding, the possibility arises that associative priming might have contributed to the effects observed by Metusalem et al. (2012). To recap, the participants in that study read short texts about common events (e.g., a baptism) containing three kinds of target words (e.g., expected, unexpected event-related, unexpected event-unrelated). The authors found that the target words in the event-related condition that were incongruent with the local sentence context yielded an attenuated N400 amplitude relative to the event-unrelated unexpected condition. It is conceivable that the LSA post-hoc analysis carried out by Metusalem and colleagues to estimate the influence of associations was not sensitive enough to capture the interplay between sentence (in)congruence and lexical-level association effects. In order to clarify the contribution of simple associations to discourse reading and in order to gain a better understanding of the underlying mechanisms, a more direct test is needed.

In the present study, we tested the potential contributions of simple word associations to discourse reading. To that end, we first replicated Metusalem et al. (2012) context manipulation (Experiment 1). Dutch participants' EEG was recorded as they read short passages consisting of three sentences. While the first two sentences established an event context, the third sentence contained one of three target words: A highly expected word, or a word that was unexpected in the context of the third sentence but related to the overall event context, or a word that was unexpected in the context of the third sentence and unrelated to the overall event. Analyses of participants' ERPs in response to the three kinds of target words closely replicated the three-way split pattern in N400 amplitude observed by Metusalem and colleagues. This demonstrated the robustness of the results in a different language than the original English study. In Experiment 2, we asked participants to read the same target sentences as before. But now, the target sentences were preceded by two sentences, which – unlike in Experiment 1 – did not build up a coherent discourse context. However, each of the two sentences contained a word (prime, hereafter) that was part of the event-establishing sentences in Experiment 1 and that was associatively related to the unexpected event-related target word in that discourse. Using this manipulation, we minimized the event knowledge that participants could extract from the two introductory sentences while keeping part of the associated lexical input the same as in Experiment 1. Thereby, we substantially reduced the possibility of participants using event knowledge to generate predictions about the upcoming target. As in Experiment 1, the primes appeared in grammatically well-formed sentences.

If the N400 amplitude reduction in the unexpected event-related condition in Experiment 1 (and in Metusalem et al., 2012, Experiment 1) was partly due to associative priming, we should observe a similar pattern of results in Experiment 2. Specifically, we predicted a significant difference between the N400 amplitude elicited by the unexpected event-related condition and the N400 amplitude elicited by the unexpected event-unrelated condition. Moreover, we predicted both unexpected conditions to differ significantly from the expected condition. Such a pattern would be consistent with the notion that simple associative relationships between words modulated the critical N400 component in the previous experiments. It would also be consistent with the notion that associations contribute to discourse comprehension. If on the other hand, the effect in the previous experiments was primarily driven by the activation of event knowledge, the event-related and event-unrelated unexpected conditions should now elicit N400 components of similar amplitude (as participants can rely less on event simulation to activate the target), with both being more negative than the N400 elicited in the expected condition.

Note that Metusalem and colleagues linked the reduced N400 amplitude in the unexpected, event-related condition to the notion that readers utilize event knowledge to predict referents that are likely to be mentioned in the described event. Similarly, many of the ERP studies reviewed above have discussed their findings in the context of predictive language comprehension. Indeed, we believe it is likely that both event simulation and associations are mechanisms contributing to predictive language processing. However, given recent methodological debates, we cannot label the behavior reported by Metusalem et al. conclusively as predictive, as the N400 component in that study was measured on the target words and not before (see Baggio and Hagoort, 2011; Huettig and Mani, 2016; Kutas and Federmeier, 2011; Mantegna et al., 2019; Nieuwland et al., 2018, for discussion). We return to this issue in the General Discussion.