Construal in language: A visual-world approach to the effects of linguistic alternations on event perception and conception

1.1 Construal: When language affects mental imagery

Construal plays a prominent role in what constitutes the core of the cognitive linguistic approach to language. For cognitive linguists, meaning resides in cognition, rather than in the relationship between language and world. While certain linguistic traditions consider the relation between language and world as a static fact that can be adequately described with truth conditions, cognitive linguists recognize that meaning is made. The language user plays an important role in making meaning as the one who negotiates the experience, its perception and its description. Therefore, meaning cannot be captured satisfactorily by an analysis of the properties of the object of conceptualization alone; instead, it requires the inclusion of the subject of conceptualization (Verhagen 2007) alongside the properties of the code used to describe the object.

Experience is so rich that there is no single way to represent a situation. The grammar of a language provides users with a range of constructions, which differ in meaning and satisfy varying semiotic and interactive goals. Construal thus (re)distributes attention in a specific way, or (re)directs attention towards certain aspects of the situation and reflects the user’s ability to adjust the focus of attention by altering what tends to be called the mental imagery associated with a situation. Both Langacker (1987) and Talmy (1988) have proposed detailed and largely overlapping classifications of construal phenomena (these classifications were later revised in Talmy 2000; Langacker 2007). Cognitive Grammar (Langacker 1987), crucially, depends on the notion of ‘profiling’, a construal operation which makes some entity stand out in profile against a background. That entity is called the Figure: it receives prominence as pivotal entity around which the scene is organized. Crucially, the Figure/Ground organization is not predetermined for a given scene and it tends to be possible to structure the scene around different Figures. Cognitive linguists have argued that language plays a pivotal role in this process as an “attention-directing device”: linguistic choices reveal and support the need to bring one element or event rather than another to the listener’s attention.

The Figure/Ground segmentation stems from research on visual perception within the tradition of Gestalt Psychology. Pioneering work on this particular problem was done by Rubin (1921), and Talmy (1978) was the first to introduce it in Cognitive linguistics. In Gestalt psychology, the whole (die Gestalt) is more than the sum of its parts, and Gestalt Formation has been invoked to explain why the mind would prefer a unit as a whole rather than its parts (Von Ehrenfels 1890). The Figure/Ground segmentation, in particular, represents the principle of perceptual grouping and, essentially, describes the process of simplifying visual input for efficient perception (cf., Wever 1927). The Figure/Ground segmentation has been fruitfully applied to interpret a wide range of language-related phenomena (Croft and Cruse 2004).

Language can be seen as a promoter or a demoter of the salience of various situational cues, which modulates how we attend to those cues. The aim of this study is to investigate the effect that the different ways to describe a scene linguistically have on that scene’s perception and conception across three types of alternations: the locational or prepositional alternation (the poster is above the bed vs the bed is below the poster), the voice alternation (the policeman arrested the thief vs the thief was arrested by the policeman) and the dative alternation (the boy gave the girl a flower vs the boy gave a flower to the girl) in English. These three alternations were chosen because they represent a cline with respect to the extent to which they implement the construal operation of perspective or prominence (Langacker 1987, Langacker 2007). Moreover, each alternation has a more frequently used variant, and a less frequently used one, with the more frequent version accounting for roughly 80% of all instances of occurrence.

The locational or prepositional alternation is a textbook example of perspective or prominence, and clearly implements the Figure/Ground idea from Gestalt Psychology. Both the poster is above the bed and the bed is below the poster denote one and the same spatial configuration but, in each case, a different element is selected as Ground with respect to which the Figure is located. In the formulation the poster is above the bed it is the bed that serves as Ground against which the Figure, i. e., the poster, is situated. But the poster serves as Ground for the bed in the bed is below the poster. In this alteration, spatial configuration is key; ultimately, the construction is about locating two entities (e. g., two objects) with respect to each other. It is typical for speakers to consider the larger entity as the Ground, and the smaller, more easily moveable entity as Figure (Shank and Walker 1989).

Voice alternations have received ample attention in linguistics and psycholinguistics (for a summary see Thompson 2012). Different from the locational alternation, the voice alternation is not about a spatial configuration but about the relations between animate agents in an event: the voice alternation thus implements a Figure/Ground distinction in a scene with animate participants. In the active voice, the policeman arrested the thief, the Agent is mentioned first and treated as Figure, while in the passive voice, the thief was arrested by the policeman, the Patient is named first and treated as Figure. In passive sentences, the Agent can remain implied or can be omitted altogether, which supports interpreting the role of the Agent as (back)ground for the Patient. It is typical for speakers to select the active construction, where the Agent is the Figure, with active sentence more frequent than passive sentences (Roland et al. 2007).

The dative alternation, finally, is the most intricate with respect to perspective or prominence as it leaves the relation between the two interacting animate beings in the scene unaffected, and instead focuses on the relation between the Object and the Recipient. Corpus analysis has suggested that the difference between the noun phrase dative (NP) the boy gave the girl a flower and the prepositional phrase dative (PP) the boy gave a flower to the girl, are determined by fine contextual differences such as giveness, animacy and pronominality (Bresnan et al. 2007). This hypothesis was later experimentally confirmed (Bresnan and Ford 2010). The NP, also known as the double object dative, is more frequent than the PP and accounts for 79% of all occurrences in a corpus of spoken language and for 62% of all occurrences in written language (Bresnan et al. 2007).

Although construal is one of the fundamental notions in the cognitive approach to language, the extent to which it invokes changes in how a scene is perceived has not yet been tested explicitly. In previous research, the conceptualization accompanying a particular linguistic choice has typically been determined by the analyst. In this paper, we rely on eye-tracking to chart how differences in describing static scenes affect the way in which language users view them.

1.2 Ception: Between perception and conception

Construal is a multi-faceted phenomenon (Croft and Cruse 2004) and several dimensions of construal can be activated in one linguistic expression (Verhagen 2007). In addition to the above-mentioned adjustments that construal makes in terms of perspective or prominence, it also affects dynamicity. Langacker’s dynamicity concerns the development of a conceptualization through processing time rather than through conceived time. It is connected to the inherent temporal nature of linguistic utterances: presenting elements of a conceptualization in a different order results in a different meaning (Verhagen 2007: 53–54). Research on sequential viewing confirms this: eye fixations follow the order of naming and elements are fixated in the order in which they will be named in a sentence just before producing a sentence (Griffin and Bock 2000; Myachykov et al. 2013).

In this paper, we continue on the path of scene viewing. In order to test to what extent a different linguistic construction of a scene by the speaker suggests a different take on that situation to the hearer, we investigate the extent to which the way in which a scene is perceived differs depending on how the scene is described. Involving perception in a study on construal is a logical step for two reasons. Involving perception is methodologically advantageous because perception can be accurately measured, as will be explained in Section 2. It is also theoretically justified, as perception and conception are conjoined within Cognitive linguistics through Talmy’s (2000) notion of ception. Talmy (2000) defined ception as a conjunction of the domains of perception and conception, representing both in a single continuous domain, to cover “all the cognitive phenomena, conscious and unconscious, understood by the conjunction of perception and conception” (p. 139). This combination of perception and conception makes ception a suitable theoretical starting point for an empirical investigation of construal, and for the following question in particular: does any difference in linguistic encoding affect the way in which events are perceived and is this effect consistent in the larger population?

To test construal behaviourally, we embed our study of construal in the larger framework of studies on visual perception and language-guided visual perception in particular. The way in which our eyes sample the environment constrains what is available for further processing (Desimone and Duncan 1995; Egeth and Yantis 1997), and the distribution of visual attention depends both on the properties of the stimulus and the observer’s goals (Bacon and Egeth 1994; Egeth and Yantis 1997; Langton et al. 2008; Parkhurst et al. 2002). That language might play a role in marking salient cues and, thus, in constraining attention has long been known. Studies investigating which elements in a static scene attracts attention revealed that (verbal) instructions affect viewing patterns (Buswell 1935; Yarbus 1967). Later research has revealed more specific properties of the language-perception link: individuals’ visual attention can be mediated by the unfolding language input (Cooper 1974; Tanenhaus et al. 1995) in that their eye movements follow the order of objects mentioned in sentences closely (Allopenna et al. 1998; Dahan et al. 2001) and anticipate objects before they are mentioned in a sentence (Altmann and Kamide 1999).

The interaction of perception and language remains a hotly debated domain of interdisciplinary research (Huettig et al. 2011; Lupyan 2012; Lupyan and Lewis 2017) and the empirical evidence that has accrued focuses on the most robust correlations between language and perception, i. e., between naming and viewing. However, the effect of more subtle linguistic differences, such as the prepositional, the voice and the dative alternations, on perception and conception remains understudied.^[1] Complementary insights are available from a psycholinguistic tradition that investigates how attentional resources are implicated in language production (Tomlin and Myachykov 2015). These studies examine how the salience of the elements in a scene and the distribution of attention over the elements in a scene influence the order in which the elements are named and the grammatical roles they are assigned in a visually situated spoken sentence across a range of different languages (e. g., English: Tomlin 1995; Russian: Myachykov and Tomlin 2008; Finnish: Myachykov et al. 2011; Korean: Hwang and Kaiser 2009).

1.3 This study

By relying on the concept of ception (Talmy 2000) to link conception to perception, we can operationalize construal in such a way that it sheds light on the mutually co-implicative domains of language (and language-encoded conceptualisation in particular) and perception. While previous findings have established a global language-perception link, the current study elucidates the linguistic specifics of that link. Using data from a visual world eye-tracking study, we investigate which differences in linguistic encoding affect scene perception across three constructions that represent a cline with respect to the extent to which they implement the construal operation of perspective or prominence. This cline is expected to be reflected both in the number of eye-movement measures that show an effect and in the strength of these effects: the preposition alternation is expected to show stronger effects across more measures, while the dative alternation is expected to show weaker effects across fewer measures. Not observing any relationship between language and perception at all would lead to the conclusion that construal is an expressive device that is informative about discourse preferences, but does not affect the distribution of attention over elements of a scene. This would limit the claims it can lay to affecting visual information uptake and hence conceptualization of a static scene.

2.1 Visual world paradigm

We used a Visual World Paradigm to investigate to what extent linguistic encoding affects the way in which events are perceived, and thus potentially conceived. The Visual World Paradigm, henceforth VWP, is an eye-tracking method often used in the context of spoken language processing (Cooper 1974). In this task, participants are presented with images on a screen, while simultaneously listening to spoken stimuli. In an attempt to understand how the linguistic description of a scene affects scene perception and conception, we opted for a consecutive processing task by introducing a slight delay between the sentence presentation offset and the image presentation onset.

2.2 Participants

Sixty students and staff (46 female; mean age = 27.4, age range: 18–57) from the University of Sheffield (UK) participated in the experiment in exchange for £7. All participants were native speakers of English and had normal or corrected-to-normal vision. Six further participants were excluded because they either failed to complete the experiment or were non-native English speakers.

2.3 Study design

We ran a cross-modal visual world eye-tracking study in which we recorded the gaze of participants as they viewed scenes and listened to a description (e. g., active or passive sentences). We used a two-level (natural image viewing vs. language-and-image viewing) within-subject design.

2.4 Data preparation

Our definition of interest areas (IAs) was content-dependent, i. e., each IA was defined empirically, based on the components of the scene. While the IAs were fixed to a scene component, the order in which the elements were mentioned changed according to the alternating construction. Details are presented in Table 1 below. Note that in Voice and Dative, A refers to the Agent, B refers to Patient or Recipient and C refers to the Action or Object. For Preposition, A captures the more easily moveable item, while B captures the less easily moveable item.

Custom IAs were created using the EyeLink Data Viewer software (SR Research, ON, Canada), and validated using fixation heat maps from 10 participants. Specifically, after aggregating heat maps for 10 participants using Data Viewer, the IAs were specified around hot fixation areas for each individual image. IAs were relevant if they corresponded to the events described in the sentences. Each image contained 2 or 3 IAs; the IA outlines were not visible to the participants. An example is provided in Image 1 for the sentence pair The policeman arrested the thief/The thief was arrested by the policeman. IA A would capture the face of the thief, IA B would cover the face of the policeman, and IA C would capture the action.

Image 1:

Fixation heat map used to determine the outline of the Interest Areas.

Data pre-processing removed all data points without eye-movements, i. e., where the first saccade entering into the IA was not available; this eliminated 33.2% of data points. An additional 1% of data points was removed because the start time of the first fixation was not available or was very short (≤60 ms, excluding 250 ms for fixation point presentation). These initial data trimming steps left us with 13,451 valid data points.

In the next step, we split the dataset into three independent (i. e., non-overlapping) datasets, one for each linguistic manipulation: Preposition, Voice, and Dative. Each of these three datasets also included the data from the corresponding naturalistic image viewing. In other words, for each participant and per image we combined data on eye-movements recorded during spontaneous image viewing, and during language-guided viewing (e. g., one for eye-movements after presenting the sentence in the active voice, and another one for eye-movements after presenting the sentence in the passive voice). The Preposition dataset consisted of 2,692 data points, the Voice dataset of 4,563, and the Dative of 5,997 data points.

Following Baayen and Milin (2010), we applied a minimal a-priori trimming strategy, removing only unambiguously discontinuous data points (i. e., those that are clear extremes, leaving a solid gap between themselves and the data mass). For the Preposition dataset this trimming resulted in an additional 0.59% of data loss. Similarly, minimal loss of data was incurred in the Voice and Dative datasets, 0.37% and 0.17% respectively. This left us with final datasets of sizes Npreposition=2669, Nvoice=4546, Ndative=5987, totalling 13,407 data points. These three datasets were subjected to statistical analyses. Appendix 1 provides summary statistics for our three main dependent variables across the three datasets.

3.1 Pre-analysing the effect of the experimental manipulation

The analyses of GazePathLength^[2] and AvgPupilSize reveal interesting trends across both models, showing an interaction of construal type (Condition) and the order in which the typical vs. atypical constructions were presented (CanonicalFirst). Figure 1 depicts the differences: (1) there is a strong within-construal consistency in the trends for both GazePathLength (downward) and AvgPupilSize (less steep and upward), and (2) there are major differences between the three types of constructions (Preposition, Voice, and Dative). From this we can conclude that there is no evidence for any differential interference effect; i. e., the general effects remain unaffected by the order in which the participants were exposed to the typical or atypical construal of an event. Thus, our experimental design did not induce effects of repetition that are different across conditions.

Figure 1:

Effects of the experimental manipulation on GazePathLength and AvgPupilSize.

Analysing the repetition in a bit more detail, we observe that the length of the gaze path shortens across experimental blocks (Figure 1, left panel). This pattern signals that participants’ visual exploration becomes more efficient: since the images are repeated across blocks, the participants can take advantage of being familiar with them. This does, however, not imply that the repetition affected proper engagement with the language-mediated conditions. The average pupil size confirms this (Figure 1, right panel): here, we observe larger pupils as the experiment unfolds, which implies that participants are working harder to integrate linguistic information with their respective static scenes. Finally, there is no evidence of task habituation or fatigue towards the end of the experiment that would negatively affect our results.

3.2 Order of access

To determine the extent to which different types of construal affect order of access, we compared the order in which the main elements in a scene were accessed, depending on the way in which they were described. Recall that the main elements in a scene constituted an IA of their own (see Section 2.4 above). The order in which each of the IAs was accessed was calculated using the time elapsed since trial onset: the start time of the first fixation in an interest area was extracted and used as first IA access time. First entrances into each IA were then rank-transformed to reflect the IA access order per participant on a given trial.

For statistical modelling we used the ordered categorical link-function which essentially assumes a latent variable following a logistic distribution, expressing the probability that the latent variable lies between certain cut-points (i. e., for the ordered categorical variable to be in the corresponding category; for details see Wood et al. 2016). For each alternation, a separate ordinal model was fitted predicting the order in which the IAs were accessed.

For Preposition, IA access order was predicted from one fixed effect, the IA label itself, and one random effect for intercept adjustments for items (images). For Voice and Dative, IA access order was, as with Prepositions, predicted from the IA itself, but the structure of the random effects became somewhat more complex. Voice and Dative further required by-participant adjustments for the location of the IAs in the scene and for CanonicalFirst. This revealed a complex pattern of (random) individual differences in how participants engage with the image after receiving specific priming from the auditorily presented description, in the particular order of construal option presentation (canonical/non-canonical or vice versa). Importantly, however, order of presentation did not contribute to predicting OrderOfAccess systematically (i. e., its parametric effect remained non-significant) for any of the models. For simple and higher-order comparisons we applied the t-test for differences between proportions, given that the predicted values are expressed as probabilities. To remain conservative, we made use of the combined standard error (as proposed by Baker and Nissim 1963) and of Bonferroni’s correction. That is, we computed the product of raw p-value and number of comparisons: p_Bonferroni = p × m, where m represents the number of comparisons (Dunn 1961). Note that we corrected for the number of theoretically justified comparisons, not the total number of possible comparisons.

Figure 2 shows that the predicted probabilities of IAs A (more moveable item) and B (less moveable item) being accessed first or second were similar across the three viewing manipulations in the Prepositional condition. Across viewing modes, the probability of the relatively more moveable element (A) to be accessed first is higher than that of the relatively less moveable element (B); this difference is significant across all three modes, as detailed in Table 2.

Figure 2:

Predicted probabilities of order of access as a function of viewing manipulation for preposition.

Second-order comparisons of the differences between the probabilities of accessing an IA sooner or later confirm these results. First, we calculated the difference between the probabilities of accessing A and B in natural viewing, typical and atypical mode (Pr_{A-B Difference}). Next, we calculated the average standard error, the second-order difference, for the two differences using their respective standard error estimates. Finally, we ran a t-test for proportions to establish whether these second-order differences between the probabilities across the three conditions were significant themselves. As shown in Table 3, the A-B differences reached significance only between naturalistic viewing and viewing in atypical mode. Overall, the typical formulation appears to align with naturalistic scene viewing, strengthening the preference for the more moveable item to be accessed first. An atypical prepositional mode neutralizes this viewing preference, as reflected in the two-fold drop, from 0.212 to 0.096, in the second-order difference (Pr_{A-B difference}).

Figure 3 illustrates the differences between the predicted probabilities of A (Agent), B (Patient) and C (Event) being accessed first, second or third for all three viewing manipulations in the Voice condition. Overall, which element is most likely to be accessed first is the inverse (mirror image) of which element is most likely to be accessed second. There is little variation in which element is accessed third across viewing modes. Details of the statistical tests are presented in Table 4.

Figure 3:

Predicted probabilities of order of access as a function of viewing manipulation for voice.

In natural viewing mode, depicted in the leftmost panel of Figure 3, the predicted probabilities of the Agent (A) or Patient (B) being accessed first are significantly lower (before Bonferroni correction), than those of the Event (C) being accessed first. A similar situation is observed in the active voice, depicted in the middle panel but here the difference is weaker. In the passive voice, depicted in the rightmost panel, the predicted probability of an element being accessed first if it is the Patient or the Event is significantly higher than if it is the Agent, while the difference between the Patient and the Event is not significant. In passive viewing mode, the probability of the Patient (B) being accessed first is also significantly higher than in natural viewing mode.

The second-order comparisons of the A-B differences, presented in Table 5, confirm these findings: the second order differences between naturalistic viewing and passive-primed conditions are significant, even after Bonferroni correction. Those between active and passive voice are likewise significant, and remain marginally significant after Bonferroni correction.

Figure 4 shows the differences between the predicted probabilities of IAs Agent (A), Recipient (B) and Object (C) being accessed first, second or last between the naturalistic and language-mediated viewing manipulations in the Dative condition; Table 6 presents the test results. In natural viewing mode, depicted in the left-most panel, the predicted probability that the Agent (A) or the Recipient (B) will be accessed first is significantly higher than for the Object (C).^[3] The Dative PP mode is depicted in the right-most panel. The predicted probabilities for the Agent (A) to be accessed first are significantly higher than those for the Recipient (B) and for the Object (C). Within the Dative NP mode, depicted in the middle panel, the probabilities of A, B and C being accessed first are not significantly different (with the difference between Agent and Recipient being marginal: p_Bonferroni = 0.095).^[4]

Figure 4:

Predicted probabilities of order of access as a function of viewing manipulation for dative.

There is no significant difference in order of access between the two dative modes on second-order comparison, as shown in Table 7. Second-order comparisons for first access between IAs B and C (Recipient and Object), which change position of mention in NP vs. PP, revealed a significant difference between naturalistic viewing and viewing after NP and after PP but not for viewing after NP vs. PP. We do again observe a decreasing trend, this time from naturalistic over NP to PP condition (Pr_{A-C difference} = 0.254, 0.033, 0.016), revealing a degree larger difference in naturalistic than in language-mediated conditions.

3.3 First run gaze duration

In order to test whether and how linguistic construal can affect and modulate which elements attract attention in a static scene, first run gaze durations were compared across viewing modes for each condition. The first run gaze duration is the summation of all fixations that occurred before the viewer moved out of an interest area for the first time. We expect the length of the first run gaze duration to reflect the extent to which the linguistic description of the event promotes or demotes elements in the scene. Separate mixed effect models were fit to each of the three datasets. The same two independent variables, ViewingMode and InterestArea, were considered as fixed effects for all three conditions but different modelling solutions were retained, as described below. All models contained two random effects: image and smooths of participants across experimental trials.

For Preposition, the model containing an interaction between ViewingMode and InterestArea turned out to be the most robust. A Wald test revealed a complex pattern of significant contrasts, visualized in Figure 5. Overall, first looks to A and to B are significantly longer in the language-mediated viewing modes than in naturalistic viewing. First looks to the more moveable item A are significantly longer after typical description than in naturalistic viewing (Chi-sq. = 31.089, p < 0.001, p_Bonferroni < 0.001) or after atypical description (Chi-sq. = 14.800, p < 0.001, p_Bonferroni < 0.001). First looks to the more moveable item A are also significantly longer than first looks to the less moveable item B following a typical description (Chi-sq. = 27.144, p < 0.001, p_Bonferroni < 0.001), but this difference is only marginally significant in naturalistic viewing (Chi-sq. = 6.129, p = 0.013, p_Bonferroni = 0.080). After an atypical description, where the more moveable item A is named last, A is looked at shorter than after a typical description (see above: Chi-sq. = 14.800, p < 0.001, p_Bonferroni < 0.001), and the less moveable item B is looked at longer than in the naturalistic viewing mode (Chi-sq. = 22.416, p < 0.001, p_Bonferroni < 0.001). Finally, across the language-mediated viewing modes and across IAs, the longest first looks are to the more moveable item A after typical description (where A is named first) and to the less moveable item B after atypical description (where B is named first), and that difference is also significant (Chi-sq. = 12.005, p < 0.001, p_Bonferroni = 0.003).

Figure 5:

First run gaze duration for Preposition.

The best model for Voice contained ViewingMode and InterestArea as main effects, without interaction; this model is depicted in Figure 6. The first run gaze duration differed significantly across all IAs within the naturalistic viewing mode (all p_Bonferroni ≤ 0.01), and within both language-mediated viewing modes (all p_Bonferroni ≤ 0.01). Within IAs but across viewing modes (naturalistic, active, and passive), differences were significant between naturalistic viewing and the language-mediated viewing modes (all p_Bonferroni < 0.001) but not between Active and Passive. Overall, the Event (C) attracted shorter first looks than the Patient (B) which attracted shorter looks than the Agent (A).

Figure 6:

First run gaze duration for Voice.

Similar to the best model for Voice, the best Dative model contained ViewingMode and InterestArea as main effects, without interaction. The findings are presented in Figure 7. Here too, the first run gaze duration differed significantly across the three IAs, within the naturalistic viewing mode and within each of the two language-mediated viewing modes. For each of the three viewing conditions, the Object (C) was fixated longest and the Recipient (B) was looked at shortest on first run (all p_Bonferroni < 0.001). The difference between first fixation durations on the Agent (A) and the Object (C) was somewhat less pronounced but remained robustly significant (all p_Bonferroni < 0.01). Yet, there were no significant differences in first run gaze duration between viewing after NP versus PP description of the situation for any of the IAs. Both language-mediated conditions (NP and PP) did require significantly longer first gaze durations than did naturalistic viewing mode (all p_Bonferroni ≤ 0.001).

Figure 7:

First run gaze duration for Dative.

3.4 Total gaze duration per interest area

A second way to establish the extent to which different types of linguistic construal modulate scene perception relies on total gaze duration per IA. Here too, more pronounced highlighting of scene components by the linguistic description is expected to extend the total gaze duration. The total gaze duration is the sum of the duration across all fixations in an IA, capturing the total length of time spent in an IA while viewing the scene. This measure is an indication of total processing effort per IA. The same two independent variables, ViewingMode and InterestArea, were considered as fixed effects for all three conditions but different solutions were retained. All models contained two random effects: image and a by-participant factorial smooth for TrialOrder.

For Preposition, the model with an interaction between ViewingMode and InterestArea turned out to be the most robust. The Wald tests revealed a complex pattern of differences. Figure 8 shows that, for each of the three viewing modes, the more moveable item A attracts the viewer’s overall gaze significantly longer than the less moveable item B (all p_Bonferroni < 0.001), but the difference between A and B is not significant after an atypical formulation (Chi-sq. = 3.7942, p = 0.051, p_Bonferroni = 0.308) when the more moveable item A is named last. More specifically, the more moveable item A is looked at longer after a typical formulation (when A is named first) than in naturalistic viewing (Chi-sq. = 7.3423, p = 0.007, p_Bonferroni = 0.040) while the less moveable item B is looked at longer after an atypical formulation (when B is named first) than in naturalistic viewing (Chi-sq. = 7.0819, p = 0.008, p_Bonferroni = 0.047). A is also looked at marginally longer after a typical than after an atypical description of the situation (Chi-sq. = 5.758, p = 0.016, p_Bonferroni = 0.098), i. e., when it is named first.

Figure 8:

Total gaze duration for Preposition.

The best model for Voice contained ViewingMode as sole fixed effect, and is therefore not presented visually. The total gaze durations differ between natural viewing versus viewing after active (Chi-sq. = 4.748, p = 0.029) and after passive (Chi-sq. = 4.406, p = 0.036) formulations. In other words, language-guided viewing modes attract the viewer’s gaze for longer overall.^[5] There is, however, no significant difference between the active and passive viewing modes (p = 1.0).

The best model for the Dative contained ViewingMode and InterestArea as main effects, without interaction. Figure 9 shows the effects that emerge. Total gaze durations for the Agent (A), Recipient (B) and Object (C) are all, pair-wise, significantly different within naturalistic viewing mode (all p_Bonferroni < 0.001), after NP dative and after PP dative (all p_Bonferroni < 0.001). NP and PP datives do not differ significantly from each other in any of the IAs, however, after Bonferroni correction.

Figure 9:

Total gaze duration for Dative.

4.1 A cline of construal

We predicted a cline in the strength of the effect that the details of the linguistic description would have on the specifics of static scene perception. We expected a pronounced effect in the prepositional alternation, a more attenuated effect in the voice alternation, and a very subtle effect for the dative alternation. The cline would show in the number of attested eye-movement measures and in the magnitude of these effects: the preposition alternation would show stronger effects across more measures, while the dative alternation would show weaker effects across fewer measures. Our findings, summarized in Table 8, confirm these expectations. Supplementary Material B contains the details of a Logistic Generalised Additive Mixed Model that confirms the existence of a cline in the strength of the effects for Order of Access.

4.2 Does language determine or modulate scene inspection?

In this paper we were interested in the question of what types of differences in linguistic encoding induce differences in visual inspection of a scene. Overall, a complex picture of effects emerges that supports the hypothesized cline from prepositional alternation, over voice alternation to dative alternation. While the prepositional alternation shows both early and late effects that mirror the order of mention, the dative alternation shows early effects of construal only, and these do not reflect the change in order of mention. The voice alternation falls in between, with early effects of change in order of mention on order of access.

The cline in the strength of the effect that linguistic construal has on perception, and by extension conception, is also visible in the obviousness or subtlety of the linguistic mechanisms involved in a given alternation. The preposition alternation, in which two objects are described in spatial relation to each other, was introduced as baseline alternation. Previous findings reported that viewers look at the elements of a scene that are mentioned in the describing sentence (Altmann and Kamide 1999; Cooper 1974), and that speakers focus on those elements in a consistent order which is, afterwards, preserved in the sentence they use to describe that same scene (Griffin and Bock 2000; Myachykov et al. 2013). In that respect, the preposition alternation, with its straightforward description of the visual scene, was most likely to show effects of construal. After all, both the typical and atypical description denote the same spatial configuration. The difference consists in the presentation of the two objects in the scene, once as ‘Figure’, another time as ‘Ground’.

In the voice alternation, the thematic roles of Agent and Patient are constant, yet variably assigned to grammatical functions. While the Agent occupies subject position and the Patient occupies object position in the active voice, their positions are switched in the passive voice and the Patient ends up in subject position, while the Agent may be omitted altogether. Hence, the choice of voice does not affect the event type, but it changes the focus on the participants in a given event. The active voice focuses on the active involvement of an Agent, while the passive voice defocuses the agentive participant in the event and refocuses on the passive involvement of the Patient. Yet, passives are only available for actives with an agentive agent, putting a semantic constraint on passive syntax (Ambridge et al. 2016; Pinker 1989).

In case of the dative alternation, much of the traditional literature has assumed that both constructions are semantically equivalent: dative shifting in English is considered a stylistic, discourse-pragmatic device (Givón 1984: 153). Goldberg (2005), on the other hand, promoted the idea that both constructions are semantically different, but statistical modelling of naturalistic data (Bresnan et al. 2007; Bresnan and Ford 2010) was needed to unravel the interplay of phonological, morphological and semantic variables that govern the probabilistic choice for one option over another. From this it can be concluded that, at the very least, any semantic differences are rather subtle. In the case of the dative alternation, construal appears to be more of an expressive device that is informative about discourse preferences but does not achieve re-focusing of the elements in a scene. This interpretation is supported by our pre-analyses (Section 3.1.) which showed that the Dative triggers the largest pupil sizes but requires only average gaze path length. In other words, the high cognitive processing costs that come with the Dative alternation are not imposed by the visual properties of the scene but by the linguistic properties of the description.

4.3 To what better end?

These findings are important for linguistic theorizing. Linguists are interested in the question of which aspects of our (linguistic) experience make it into (long-term) memory. Our experience is far too rich for us to remember every detail, and between experience and memory lies a process of attention and encoding, which makes memory not directly reflective of frequency (Pierrehumbert 2006) or active/passive decay of memory traces (cf., Hebb 1949; Underwood 1957; for recent findings and discussion see Mirković et al. 2019). The results we have obtained here shed light on the kinds of differences in linguistic encoding that affect attention for a specific aspect of a scene, thereby increasing the chances for those aspects of experience to leave traces in memory and to become gradually more entrenched. Language, in other words, reduces the uncertainty about which components of a scene are important and, eventually, which ones should be remembered.

Given the set-up of our study, we are looking at the very first stages of memory, but in order for something to make it into long term memory, it will need to have been retained at this first stage. As explained in Section 2, the average auditory stimulus duration was 2600 ms (range: 1877–3672 ms). The auditory stimulus was followed by a brief 250 ms fixation on a central point, before the image appeared and remained visible for 3500 ms. On average, 60 ms were needed from the time the image appeared before the eyes fixated for the first time. Although a stimulus can persist in auditory memory for up to 3 or 4 seconds, in most cases, only the end of the stimulus will have been available in echoic memory by the time the participants started to view the scene. Thus, at late viewing time (~ 6000 ms) participants will have been relying on processed information available in working memory only. As reported in Section 3 and discussed above in Section 4.1, not all linguistic manipulations are equally successful in affecting attention, and we observed early and late effects of linguistic manipulation on visual information uptake. While the prepositional alternation shows both early and late effects that clearly mirror the linguistic manipulation, the voice alternation shows early effects of construal on order of access only and the dative alternation shows some early effects that do not map straightforwardly onto the details of the linguistic manipulation.

The strength of the effect that linguistic description has on scene perception determines the claims language can lay to affecting visual information uptake and hence conceptualization of a static scene (Huettig et al. 2011; Lupyan 2012; Lupyan and Lewis 2017). In the case of Preposition, the effects of language on perception were clear and strong. It seems convincing to assume that the two reinforce each other mutually, and work together during information uptake and processing. Language guides attention, thereby influencing how a scene is perceived, and shaping the experience that is committed to memory. In the case of the Dative, the effects of language on perception were weak and indistinct. It is, thus, much less plausible to postulate some (or any) level of mutual reinforcement. Here, language modulates attention by highlighting aspects of a scene, while not fundamentally changing how the scene is perceived and committed to memory. The linguistic preferences, although entrenched in memory (Bresnan and Ford 2010), remain restricted to the level of the code, as some sort of ‘larpurlartistic’ device that, essentially, does not alter the uncertainty of the message (by either increasing or decreasing it substantially) but, perhaps, serves to build plasticity of the communicative system – the language.