Chunking or predicting – frequency information and reduction in the perception of multi-word sequences

2.1 Participants and stimuli

The data come from 38 native speakers of American English (20 female), who were offered a compensation for their participation. The stimuli consist of 126 recorded sentences in American English, all produced by the same speaker. These comprise 42 target items containing a V-to-V_inf construction, each with a different verb before to (1); 42 control items containing to in a different construction (2); and 42 distractors which do not contain to at all (3). The target items were designed to ensure that the subject was always a pronoun and the target word to occurred in the middle of the sentence. Control items contained the same verbs as the target items, but not in a V-to-V_inf sequence, so that participants could not simply assume that a verb would be followed by to whenever plausible.

1. When the penguins are around, we pretend to like the way they dress.

2. When the monkeys come over to dance, I pretend I’m asleep.

3. I can’t believe the crocodiles are selling leather handbags.

Each target item was recorded in two conditions: “full” and “reduced”, referring to the pronunciation of to. Full pronunciations consisted of a full [t] and a short [ʊ]; reduced forms had a flap [ɾ] and a schwa [ə] (e.g., pretend to as “pretenda”). The recordings were checked to make sure that to could be distinguished as full or lenited on the basis of both auditory impression and acoustic analysis (waveform). ^[2] Speaking rate in the test sentences was held constant at around 5 syllables per second. ^[3]

Each participant responded to 126 items in total, 42 each of target, control and distractor items. The target items all contained different instances of V-to-V_inf, 21 of which were presented in the “reduced” condition and 21 as full forms. Participants were assigned to one of two groups, so that group A would hear full forms of those items that were reduced for group B, and vice versa (see Appendix for a list of the items). The control items were the same for each group and all presented with a full to.

2.2 Design and procedure

Participants were asked to respond to the presence or absence of to as accurately and quickly as possible by pressing a key on the keyboard. They were told that “to is a very common and versatile word in English. It can serve many functions, and it is not always pronounced the same way”. The experiment was preceded by a brief practice phase. Participants answered a simple comprehension question on 10 of the control items during the experiment to confirm their continuing attention to the stimuli. The order of the stimuli was randomized.

The software OpenSesame (version 3.0.7 for Mac, Mathôt et al. 2012) was used to carry out the experiment and record response times. By using the “xpyriment” backend, the experiment runs on the Python library Expyriment (Krause and Lindemann 2014), which is recommended due to its timing precision. The “prepare-run strategy” implemented in OpenSesame (i.e., preparing a stimulus before running it) further prevents timing errors due to other hardware operations. Since the experiment was run on different computers in different locations, the temporal jitter was tested on each machine by 30 random trials; the deviations did not exceed +/-10 milliseconds.

2.3 Variables and analysis

3.1 Accuracy

The overall accuracy rate on the 1,596 target items is 85.7%. Full renderings of V-to-V_inf were recognized more consistently (accuracy 92.4%) than the reduced variants (78.9%). On control items, the accuracy rate is 88.6%. Thus, as expected, full target items do not differ significantly from control items, while reduced forms produce more errors.

To see whether this difference is influenced by frequency, conditional probability or other variables, a binomial generalized additive model (GAM) was fitted following the procedure set out above. The resulting model includes smooth terms for the test variables frequency (“logfreq”) and transitional probability (“TP”); for “TP”, the interaction with “condition” turned out to be irrelevant and is excluded from the model. Fixed effects are included for three control variables: “plosive cluster”, i.e., a final /t/ or /d/ in the verb preceding to; “control before target”, i.e., whether the verb had already been heard in a control item; and the item count during the experiment (included as a linear effect because a smooth term provided no clear advantage to the model). Random effects are present as intercept adjustments for “subject” and “verb” (the test item). The model is specified as

The model provides a good fit to the data (C = 0.836, UBRE = –0.301) ^[6]; it is presented in Table 1 and visualized in Figure 1. ^[7] It should be noted that since accuracy is fairly high overall, the results it provides are rather coarse-grained.

Figure 1:

Effects on accuracy. Y-axes for smooth terms (upper row) refer to estimated effects, which are centered and scaled by the GAM. Y-axes for parametric terms (lower row) refer to model predictions of error rate.

According to the model, frequency and transitional probability have little impact on the occurrence of errors. There are slightly fewer recognition errors with high TP, but this trend is not significant (p = 0.117). Frequency is marginally significant only on reduced items (p = 0.057). While it should be viewed with caution, the smooth curve (Figure 1, top middle) suggests that items of mid-high frequency are recognized with the fewest errors. Moreover, the presence of a plosive merger increases the error rate on reduced items (Figure 1, bottom left).

These first results already hint at a possible facilitating role of probability; the difficulty of recognizing reduced forms is aggravated when reduction affects a morpheme boundary (as with plosive clusters), and may be mitigated at mid-high levels of frequency and high TP. Measuring accuracy can only provide a rough measure of these effects, and the smooth terms only yield statistical trends. It will be seen that similar effects hold more clearly with the response times on correctly recognized items. In addition, participants made fewer errors when they had heard the same verb in a different construction (i.e., a control item) before the V-to-V_inf item; error rates on reduced forms in particular decrease in the course of the experiment (“item count”). This suggests that there are learning effects as participants get attuned to the voice and content of the stimuli. While the learning effect applies to accuracy, we will see that it does not translate into response times.

3.2 Response times

The overall mean response time was 675 ms (median: 580 ms). As the bean plots in Figure 2 show, responses to full renderings of V-to-V_inf are just slightly faster than to control items (t = –1.997, p = 0.046), whereas a reduced to elicits considerably longer response times (t = 8.924 p < 0.001). ^[8] Thus, as expected, reduction of the target word generally delays its recognition. The crucial question is to what extent this delay may be mitigated by frequency, conditional probability or other, phonological cues.

Figure 2:

Response times to control, full and reduced items.

We fitted the log-transformed response times of target items (n = 1,367) to a mixed-effects generalized additive model following the procedure set out above. With “condition” (full vs reduced) as a moderator variable, the model tests how the effect of reduction is affected by the other variables. The model includes smooth terms for the test variables “frequency” (log-transformed) and “transitional probability”, ^[9] and parametric terms for the control variables “verb duration”, “plosive cluster”, “age” and “gender”. Random intercepts are included for “subject” and “verb” as well as random smooths for item count (by subject) and condition (by verb). The final model is thus defined as:

Overly influential outliers were eliminated post-hoc through the model residuals, removing data points with a residual value beyond 2.5 standard deviations (33 items; cf. Baayen and Milin 2010: 17–18). After this trimming, the final model stands up to scrutiny (near-normal distribution and constant variance of residuals, no extreme outliers) and reaches R²= 0.612. ^[10] It is presented in Table 2.

3.2.1 Test variables

In the model in Table 2, “condition” shows the expected effect of reduced forms leading to longer response times. The smooths in Figure 3 show how the effects of frequency (“logfreq”) and “TP” also vary between conditions.

Figure 3:

Effects of surface frequency and transitional probability on response times.

For frequency, the smooth terms in the model show that there is a strong (and linear) effect on the full forms, while the effect on reduced forms barely reaches significance. As the curves in Figure 3 (upper panels) show, low-frequency items produce longer response latencies in both conditions; recognition of full forms continuously profits from higher frequency, producing a linear effect (edf = 1, p < 0.001). Response times to reduced forms rather follow a shallow U-shaped curve, which verges on statistical significance (edf = 3.82, p = 0.055). Higher frequencies lead to faster responses up to a point at around logfreq = 3, but this trend does not continue at very high frequencies.

For transitional probability, there is a very clear interaction with “condition” (Table 2). An increase in TP does not make a significant difference to recognition of to in its full form, but it has a clear impact on the reduced items (p < 0.001). The effect is linear (edf = 1), such that recognition of reduced forms is continuously faster with increasing TP.

The differences in recognition latencies between full and reduced forms at different frequencies and TPs are shown in Figure 4. Here, the response times are plotted as estimated by the model for the interactions of “condition” with “frequency” (left) and “TP” (right), with all other effects (of control and random variables) held at their mean. ^[11] Confidence bands refer to 1.96 standard errors of the effect of “frequency” or “TP”, respectively.

Figure 4:

Estimated response times by “condition” and “log frequency” (left)/“TP” (right).

The left panel shows that, expectedly, reduced forms produce slower responses across the board. The curves for the two conditions follow parallel developments up to a point, showing a facilitating effect of frequency. At higher frequencies, this trend continues with full forms while responses to reduced forms are somewhat delayed. A “chunking effect” of high frequency is thus observed for reduced forms only, although this effect is not very clear (with p = 0.055 for the smooth term, see Table 2). What is clear, however, is that the gap between the full and reduced variants widens at high frequencies.

The right panel shows again that recognition of reduced forms profits from higher TP, while full forms are not affected. The two linear curves overlap at high TPs, where the difference is almost levelled; it seems that reduction does not, or scarcely, delay recognition when the item is highly predictable from the immediate context.

3.2.2 Effect of control variables

Since our original hypothesis concerned frequency and conditional probability, other properties of the input items are considered control variables. Yet, these control variables turn out to play an important role. Controlling for age reveals that older participants show somewhat slower responses, mainly on the full forms (upper left panel in Figure 5). It looks as though they did the task a little more cautiously rather than reacting quickly (they also responded slower to control items); note though, that the “older” end of the scale is represented only by few participants. Regarding gender, male participants show a greater delay with reduced forms (upper right panel). This effect is weak in the model (t = 1.89, p = 0.06), and while it has to be acknowledged, any far-reaching interpretation would seem out of place. In sum, there is a tendency for younger and male participants, respectively, to show a greater difference in their responses to full and reduced.

Figure 5:

Response times to full and reduced items by control variables.

The other control factors that remain in the model (making a significant contribution to explaining the results) are those that measure properties of the verb preceding to, i.e., its duration and the phonological segments at its boundary. Responses to to are faster when the preceding verb has a longer duration, regardless of condition (lower left panel in Figure 5). A longer verb provides a longer span for its processing; this prompts a faster recognition of the upcoming item. Finally, when the final segment of the verb preceding to is an alveolar plosive (e.g., need to, hate to), the cluster ([dt] or [tt]) gets merged into a single segment – this case inhibits recognition of a reduced to but has no effect on full forms (lower right panel). The merger produces an effect of coalescence as it obscures a word boundary. This effect is discussed further in Section 4.3.

4.1 Frequency information

The main question we asked was how frequency information interacts with reduction in speech perception – do listeners access and use this information in coping with reduced input?

For the surface frequency of the sequence (V + to), we hypothesized that high frequencies would help the recognition of reduced forms, since frequent sequences form an entrenched routine, and reduction is associated with frequency in speech production; we further expected to see a chunking effect delaying recognition of both full and reduced forms at very high frequencies (cf. Sosa and MacFarlane 2002; Kapatsinski and Radicke 2009).

The results we obtain for string frequency only partially match these expectations (see Figure 4, left panel). The linear effect of frequency on full forms (faster responses to items of higher frequency) suggests that there is a general facilitative role of string frequency in word recognition. This would imply that frequency simply serves as a base probability (cf. Hall et al. 2018: 2) – less frequent items are less expected, more frequent items are more expected – irrespective of the items’ compositionality and their potential chunking. What frequency facilitates, then, is the processing of separate items in a sequence. Recognition of reduced forms, however, is aided by frequency only up to a point (around log frequency = 3). If there is a frequency range in which listeners may expect reduction and therefore identify a reduced item more easily, this would be at these mid-high frequencies. Higher frequencies facilitate recognition only of the full forms but provide no further advantage with reduced forms. This suggests that if chunking interferes with accessing the element to (and offsets the facilitating frequency effect), it does so only when to is reduced. In other words, the activation of a holistic representation is based not only on frequency, but also on the phonetic form of the input.

For conditional probability, we expected that when to is highly probable (given the previous word), it will be recognized faster in general and perhaps especially aid recognition of reduced items. However, the results (Figure 4, right panel) show that recognition of full forms is insensitive to TP, suggesting that a higher probability has no general facilitating effect. Reduced forms, in contrast, show a strong linear effect, with higher TP continuously facilitating recognition. As transitional probability refers to predictability from context, it seems that listeners benefit from this information when it is most useful, in this case to recover a reduced form. At low TPs, to is least predictable from context, and it is here that reduction causes the greatest difficulty. At high TPs, to is more easily recognized in spite of reduction. If listeners draw on probability to derive a strong expectation of an upcoming to, they need less information from the input to identify it. On the other hand, when the full phonetic cue is available (i.e., full forms), it seems that listeners do not need to resort to the information from transitional probability as a further support in recognizing the item. One might suspect that the lack of an effect of TP with full forms is due to a floor effect on response times: when the signal is very clear (i.e., full forms), there is simply less room for listener expectations to improve recognition. However, responses to full forms are not consistently “fast”, as can be seen from the strong effect of frequency on full forms (where response times to low frequency items are at ca. 600–750 ms, high frequency at 400–500 ms). This variance would, in principle, allow for an effect of TP as well, but this is not found.

It is noteworthy that we find strong effects of forward transitional probability (the probability of to given the preceding word), but not of backward transitional probability (the probability of to given the following word) or the surface frequency of the to-V_inf sequence. This may be due to the restricted set of fairly common verbs following to in our stimuli, which were designed to focus on the preceding rather than the following verb. In speech production, several corpus studies find measures involving the following word to have an impact, e.g., on duration and vowel reduction in function words such as to (Jurafsky et al. 2001; Bell et al. 2003). However, Bell et al. (2009) report this effect to hold only in mid- and low-frequency function words, whereas high-frequency items (such as to) are conditioned only by forward probability. They propose that high-frequency function words “often include alternate forms lacking onsets for the words with obstruent onsets (the, that, to). This makes them vulnerable to reduction when closely associated with previous words” (Bell et al. 2009: 108). Since our design included lenition of the /t/ onset of to, it appears that this association with previous words in production translates to higher expectations of reduction in perception. We cannot make a conclusive statement about any similar association with the following word.

4.2 Effect of phonological context

In addition to the effects of frequency and transitional probability, we observe a strong influence of phonological properties of the verb preceding to.

A longer duration of the verb accelerates recognition of a following to (see Figure 5, lower left panel). It seems that this is merely a side-effect, as longer words (e.g., remember) can be recognized before they are completed and thus allow listeners to more quickly predict and process the next item.

The inhibitory effect of reduction is much greater when /t/ merges with a preceding alveolar plosive (e.g., in need to, pretend to; Figure 5, lower right panel). Given that listeners are sensitive to fine-grained phonetic cues to identify word boundaries (cf. Fernandes et al. 2007), we can interpret this as an effect of on-line phonetic coalescence: when such a merged plosive is lenited, it marks a reduction not only at the onset of to but also at the offset of the preceding verb and thus blurs the word boundary. This creates a stronger connection (coalescence) between the verb and to, and may lead listeners to a non-compositional access path. The delay in recognition is then caused by the need to decompose the unit (V + to) in order to identify the element to. It would seem that this coalescence in phonetic realization has a similar effect to frequency-based chunking in fusing two elements together. Indeed, in natural speech production, phonetic coalescence and frequency-based chunking are probably interconnected in the production-perception loop: a coalesced realization (e.g., “needa”) will reinforce a non-compositional mental representation, which will in turn increase the frequency of reduced realizations. Thus, a reduced merger of alveolar plosives seems favorable to the propagation of a single-unit representation of a high frequency sequence (need to). ^[12]

However, frequency-based chunking is seen as holistic storage in memory, so that the “chunk” can be accessed as an item, whereas on-line phonetic coalescence is a matter of articulation in speech production and needs to be decoded on the spot. The delayed responses to strongly coalesced realizations (reduction in the merged plosive, e.g., “needa”, “pretenda”) thus point to the gradient nature of chunking (cf. Ellis 2002). The fusion of two elements is represented in memory to varying degrees, and its activation depends on the form of the input. When the word onset is not clearly identifiable, listeners have greater difficulties in singling out the word to as separate from the previous word. This finding lines up with accounts that word-initial material has special importance in word recognition (Marslen-Wilson and Tyler 1980; McQueen et al. 2003; Astheimer and Sanders 2011; but see Connine et al. 1993).

4.3 Prediction, entrenchment, chunking

Our broad finding has been that reduction delays the recognition of an item, that frequency information can aid recognition, and that forward transitional probability is most clearly instrumental in recovering reduced forms. We also find a modest “chunking effect” of high frequency, albeit only on reduced forms. Thus, the activation of a chunk – which might interfere with listeners’ recognition of an individual element – depends not only on frequency but also on the phonetic form of the input (i.e., reduced). Moreover, reduction that obscures the word boundary (“plosive cluster” in our results) inhibits recognition, suggesting a connection between chunking and on-line phonetic coalescence. We now turn to a discussion of these results in light of relevant concepts of cognitive linguistics, in particular prediction, entrenchment and chunking.

Frequencies of multi-word sequences are known to affect speech production (Arnon and Cohen Priva 2013; Gradoville 2017) – our results confirm that frequency also plays a role in speech perception (see also Van de Ven and Ernestus 2018), mainly in that low bigram frequency causes greater difficulty for word recognition. However, speech perception may also rely on predictability and probabilistic cues. Listeners may intuitively employ probabilistic measures related to the previous input in order to form expectations and anticipate the next word. Moreover, they draw on this information especially when the present input is difficult to recognize by itself (Ernestus et al. 2002; Mattys et al. 2012), so that prediction could be seen as a compensation strategy (cf. Pickering and Garrod 2007).

The present results suggest that frequency-based probabilities (derived from surface frequency and transitional probability) are available to listeners as an aid for anticipating upcoming words. This requires, minimally, a tacit knowledge of such probabilities. ^[13] While surface frequency shows an effect with both full and reduced inputs, transitional probability is relied on more in the case of reduction, i.e., when phonetic cues are weaker. This is in line with other accounts of probabilistic prediction. Firstly, there is evidence in speech processing that conditional probabilities are less important than other, more direct cues (lexical information, phonological matching; Mattys et al. 2005; Franco and Destrebecqz 2012). Huettig and Mani (2016: 19) argue that probabilistic prediction “provides a ‘helping hand’ but is not necessary for language processing”. Secondly, the degree to which listeners reach for this “helping hand” is dependent on its utility given the task and goals at hand (Kuperberg and Jaeger 2016: 44–45). In a word monitoring task, this utility is presumably very high at any point. The results of the present experiment suggest that the utility of (probabilistic) frequency information is especially relevant when the acoustic information is less complete (as in reduction).

Another current issue in cognitive linguistics concerns the mental representation of multi-word sequences, namely the tension between holistic storage (chunking) and procedure strengthening as two different possible interpretations of the notion of entrenchment: are highly frequent phrases stored as single units (Bybee 2006; Diessel 2007; Ellis et al. 2009), or are their pieces more rapidly assembled due to continual usage (Divjak and Caldwell-Harris 2015: 66–67)? Speech-monitoring studies have provided partial evidence for faster retrieval of elements from more frequent bigrams, which is offset by a chunking effect at (very) high bigram frequency (Kapatsinski and Radicke 2009). The present study contributes to this research by putting reduction into the picture, which differentiates these effects. On the one hand, procedure strengthening is seen in the continuous facilitating effect of frequency on full forms and the capacity of transitional probability to balance out the adverse effect of reduction. On the other hand, a modest chunking effect of frequency is found for reduced items, in the delayed response times at high frequencies.

Procedure strengthening and chunking may be conceived of as stages of a continuous process (cf. Langacker 1987: 59–60; Blumenthal-Dramé 2012: 68–69, 104). With increasing frequency, a sequence is increasingly entrenched as a procedure, and with increasing entrenchment, boundaries between the individual components become less important. The end result is a chunk, a mental representation as a single unit. However, the present results suggest that even with very high frequency, chunking is not inevitable. Listeners may perceive a bigram as either a chunked item or a compositional sequence, i.e., they have two access paths available, that of a memorized chunk and that of a compositional sequence. The activation of one or the other access path is affected not just by frequency but by the properties of the input signal. When reduced forms occur in a high-frequency bigram (e.g., “needa” need to, “havda” have to), they lead the listener into perceiving a chunked item. Full forms on the other hand encourage a compositional access. In short, for high-frequency bigrams, the presence or absence of reduction determines whether listeners perceive a chunked item or a compositional sequence.

This conclusion is not compatible with a one-dimensional concept of entrenchment, where at some point of frequency a sequence becomes reanalyzed as a chunk. Rather, we need a “continuous” approach in which “the difference between more and less frequent is […] one of degree, rather than specifying whether the sequence is stored vs. computed” (Caldwell-Harris et al. 2012: 3–4), and, crucially, a concept of chunking as “global precedence”: “a configuration of elements qualifies as a cognitive chunk, if the configuration as a whole is cognitively more prominent than its individual component parts” (Blumenthal-Dramé 2018: 138). In this view, there is a mental representation of the “whole”, i.e., a stored chunk, but what is activated in perception – whole or parts – is a matter of prominence. “Cognitive prominence” may be seen as a kind of base activation of the holistic representation; but to explain the discrepancy in the frequency effects on full and reduced forms, we must add a perceptual prominence. With reduction, the individual parts are made less prominent, so a perception of the “whole” – if accessible as a mental representation – takes over. With full forms, the individual items are clearly distinguishable and can therefore be accessed without interference from the holistic representation.

The finding that the chunking effect is limited to reduced forms, may also be interpreted in terms of stored pronunciation variants. For single words, listeners are responsive to pronunciation variants, and even to the correlation between word frequency and phonetic reduction (Ranbom and Connine 2007; Bürki et al. 2011; Mitterer and Russell 2013; Brand and Ernestus 2018). If high-frequency sequences (e.g., need to) are liable to phonetic reduction (e.g., “needa”), these reduced forms may also be stored as variants. In the experiment, it is possible that reduction of to in very high frequency “V-to” sequences activates these variant representations. These would be variants of a single unit in which the original elements are less prominent. The mental representation of a chunked item will then delay the recognition of a reduced element (in this case to) because it also includes reduced variant forms (e.g., needa).

Finally, there is the special case where reduction occurs at a merged word boundary (e.g., “needa” need to, but also “hayda” hate to). The observed recognition delay suggests that listeners have to deal with an additional limitation that is somehow related to chunking, but probably not equivalent: in frequency-based chunking the processing difficulty is caused by the interference of an item stored in memory; however, a reduced word boundary can be a matter of on-line phonetic coalescence in speech production and simply needs to be decoded by the listener on the spot. Further research could elucidate the exact relation between on-line phonetic coalescence and memorized chunks.