Narratives of the self in bilingual speakers: the neurophenomenal space

1.1 Preliminaries

As a way to make the matter at issue more concrete, consider talking about an experience of visiting a new city. In response to a question about what it was like from a person you don’t know well, you reply: “Interesting, English-speaking, some nice restaurants and bars”. But what is felt and not expressed? Walking through a city unfolds diverse landscapes: linguistic landscapes that tell a story and impact on the lives of citizens (e.g., Ben-Rafael et al. 2008; Wei and Hua 2021), linguistic soundscapes (El Ayadi 2022) where overhearing speakers of different languages can trigger impromtu conversations (Hänggi 2022) and memories of other times and places that are rich in experiential detail (see Pink 2008 for implications for ethnographic research and Cabeza and St. Jacques 2007 for a characterization of the neural bases of such memories). Further, given that we are embodied beings, walking through a city yields a sensory, kinaesthetic and olfactory map of the experience (te Broemmelstroet et al. 2017). Language acts to bind us to our multidimensional experiences and allows us to share aspect of those experiences when we talk with one another. Unspoken passages of thought, feeling and bodily experiences though will also be reflected in the neural dynamics of a person’s speech and so we need ways for the speaker to express these too. In order to map between the subjective experience and the neural substrate we need to consider the processes that give rise to the experience. It is interpreting and amplifying these processes that permit the mapping between the subjective, phenomenal, space and its neural substrate. I consider this matter in the next two sections.

1.2 A conceptual sketch

As we talk we share our lived experience. We can do so because our subjective experiences have much in common. Our experience of the world is multimodal, integrated and embodied. Our experience has perspectival properties. These allow us to envisage other points of view and are a vital ground for our ability to attribute psychological states to one another (e.g., Wiliford et al. 2018). Such attribution is essential for understanding the stories we share. And it is the psychological properties of our phenomenal experience rather than its perspectival properties per se that I want to consider. This distinction is evident in our everyday experience as we can walk together and readily talk about matters unrelated to the immediate environment that we are walking in.

I start from the view that our subjective experience is an interpretation of the world (see Ramstead et al. 2022 for a computational approach aligned to this view). It is an interpretation that is richly informed by our evolutionary past and by our cultural worlds. In conversation there is an issue of current concern for us – the story we want to tell. It is a focus that renders other potential matters of interest to the periphery yielding a structure to our phenomenal experience.

Given its cognitive and affective qualities I term it the cognitive-affective nexus. Its qualities drive our talk.

Conceptually, we can view the mapping between the subjective and neural as embedded within an interface between ourselves and the world. This interface mediates a two-way relation between our internal (interoceptive) and external (exteroceptive) worlds. It includes the motor and auditory processes involved in speech production and comprehension and also mediates our sensory and kinaesthetic experiences and actions in the world more generally. Our subjective experience arises from the interface but also from our stored experiences and the mental actions including inferences and attributions we make as we talk about a particular issue. Figure 1 offers a conceptual sketch of the neurophenomenal space. I identify the neurophenomenal space as a mediated mapping between the phenomenal and the neural.

Figure 1:

Conceptual sketch of the neurophenomenal space (please see the text for explanation).

In Figure 1, the surrounding interface (a blue enclosing space or membrane) is open to reveal mappings (yellow lines) between the phenomenal state (top part of the Figure) with its focus, the cognitive-affective nexus (inner ellipse, purple) and periphery (pink ellipse) via the middle part of the Figure, with its attentional, cognitive-affective, and linguistic processes, to the neural bases that index embedded neural states. The middle part of the Figure schematises thinking for speaking (Slobin 1996). Under command of the attentional processes (closed filled ellipses) and meta-attentional process (closed filled ellipse within an ellipse) the intention of a speech act (light blue ellipse, right side of the figure) progressively recruits and maps material from experiential and inferential domains (networked open ellipses) to distributed regions of the language network (overlapping open ellipses) in accord with the intended task (e.g., to narrate a story) and language control processes (green ellipse). At the neural base, triangular peaks connote variations in demand on attentional networks and regions contingent on cognitive, affective and linguistic demands in the dynamical flow of neural activity.

Attentional processes and the states they induce shape the phenomenal content of the nexus and so are integral to exploring its neural bases. Manifestly, phenomenal content differs markedly when we are awake than when we sleep but even when we are awake and attentive to some task our minds may wander. Fortunately, as we know, our current state is also one we can monitor and adjust – we can get back on track (as per the meta-attentional processes noted above). The upshot of these facts is that in theorising about the neurophenomenal space we need to recognize that our phenomenal experience reflects a basic aspect of our lived experience: we have a degree of command over our attentional states and their properties. I consider the attentional states involved in narrating a story next.

1.3 Attentional and mental states involved in narrating a story

In my illustrative study, my hypothetical speaker (JM) tells of a recent experience of a walk in a new city with his new boss. A fundamental presumption guides my identification of the invoked passages of thought and feeling with their putative neural substrate. I presume that as we tell a story of a personal experience we relive it. Our reliving is a simulation – a virtual reconstruction-that calls on cognitive and affective aspects of the original experience. Are there grounds for such a presumption? Reliving is involved in retrieving autobiographical memories (Cabeza and St. Jaques 2007) and is usefully conceptualized as a form of mental simulation (Kent and Lamberts 2007). To my knowledge, most neurocognitive research on narratives concerns how listeners respond to another person’s narrative and does not concern the experience of the teller of their own personal story. And yet there is a deep connection between these two activities. As we listen to a story or read a work of fiction we are actively simulating what is going on in the minds of the participants and considering what else might have happened or could happen. Such simulation underlies our felt immersion in the world of the story (Oatley 2016). And, fortunately, we do know that a listener’s degree of immersion in the world of the story or movie significantly affects their neural response as they listen to it (e.g., Song et al. 2021; Vaccaro et al. 2021; see also Dini et al. 2023 with respect to film clips). If that is true of our response to another’s story, we can be sure that felt immersion is greater for its teller. I contend then that telling a personal story is justifiably treated as simulated reliving of the events and personal actions involved. Ultimately though I rely on our own subjective experience for its truth.

Telling a personal story then requires speakers to sustain attention to the telling of it and also requires sustained mental simulation to allow the story to unfold. The latter process may be viewed as being nested within the former. With respect to nested attentional states, I take such states as ubiquitous of our everyday experience in the world. For example, on the way to meet up with friends we may think about other matters of concern. A superordinate state calls on other states and while it is currently active (its goal is being maintained) component states are nested within it. Experimental research necessarily involves nested attentional states. We agree to participate in an experiment but our lives aren’t simply suspended. The phenomenon of mind-wandering is indicative. Consider an experimental task that requires us to identify a specific signal and press a button. How do I realize I’ve drifted off task unless I’m monitoring my current state with respect to the goal? Clearly, I can be more or less engaged and that is reflected in my phenomenal state. Indeed, work requiring participants to report on the nature of their mind-wandering during such an experiment reveals a range of hidden attentional states (using Markov-chain modelling) that are predictive of performance (Zanesco et al. 2020). However, what is overlooked, is that the attentional demands of the entire experiment are nested with the wider concerns of the participant. The nested states considered in storytelling are different – they are interrelated. The goal is the telling of a personal story and during its unfolding other states (e.g., more distant autobiographical memories) are evoked as part of the mental simulation.

Stories also express inferences about our own and others’ states of mind. I use the standard term ‘mentalizing’ to cover inferences about the thoughts and feelings of oneself and others and also to cover the emotional assessment of oneself and others (see Henry et al. 2021). Mentalizing processes then are part of the simulation and may be thought of as nested within it (see Green 2019 for elaboration within the broader context of conversation in general). And finally, of course, telling a story involves language use. Subjective states are realised and expressed through the language network entailing their binding to the processes of speech production. Utterances are also designed to be understood by the addressee and the addressee assumes that to be the case too (Clark and Murphy 1982). For a bilingual speaker, where only one of the story teller’s language can be understood by the addressee then only that one can used with the other gated from their speech. Switching between languages, and between different types of codeswitching, require different settings of the processes for language control (Green 2018; Green and Wei 2014). In the next section (2) I describe two methods for characterizing subjective experience and then in Section 3 discuss their use in the illustrative study with JM. In the final section (4) I review the proposal and consider some prospects.

2.1 Path diagrams

The cognitive-affective nexus has a structure expressing the issue of concern. Path diagrams can elicit that structure. In a path diagram, paths are drawn to connect the key forces at play as perceived by the story teller. Such diagrams have been used before to capture an individual’s understanding of a particular phenomenon such as the factors bearing on the risk and prevention of coronary heart disease (Green and McManus 1995) and the causes of overfishing (Nikolic and Lagnado 2015) and have been used in formal analyses to identify the causal bases of various phenomena (e.g., Pearl and MacKenzie 2018). The novelty is to use them to capture the structure of the cognitive-affective nexus. In this case, the paths reflect a person’s understanding of an experience. And that understanding is based on their interactions with the physical and interpersonal worlds informed by their languages and their socio-cultural values and beliefs.

Green (2022) illustrated the use of path diagrams to express the communicative rupture between a young person and her parents. Her parents had migrated to America some thirty years previously and only spoke Cantonese. A first diagram captured the sources of her pain and regret. A second diagram captured the transformative effect of two interventions she proposed: to relearn Cantonese and to construct a new Cantonese-American identity that acknowledged her Cantonese heritage. The difference between the two diagrams (these snapshots of her phenomenal state at two distinct time points) expressed the transformative phenomenal flow.

As regards relating the phenomenal states depicted in the path diagrams to their neural substrates we can be guided by research that has looked at the networks recruited as individuals listen to stories (Song et al. 2021); by research on autobiographical memory (e.g., Cabeza and St. Jacques 2007); and by research on mentalizing (e.g., Henry et al. 2021).

2.2 Temporal experience tracing

As with path diagrams, the technique is intuitive. Participants trace (draw), on a piece of graph paper the felt intensity of their experience on a number of dimensions over time. Jachs (2021) used the technique with different groups of meditators after they had completed a meditation. Her dimensions were familiar to the meditators and representative of the demands and goals of the particular meditational technique. They included ones familiar in other contexts that require sustained attention with dimensions such as the amount of effort and boredom. Other dimensions were unique to a particular technique such as “stillness” in Zen meditation.

Jachs (2021) showed that the temporal experience traces, when digitised, related to the EEG time series recorded as they meditated and so might be informative of the neural bases of phenomenal states elicited as a person talks of an experience.

3.1 JM’s story

JM says he’d like to talk about his first meeting with his new boss. The gist of his story is that his new boss was interesting and informative about the streets and sights as they walked along to meet JM’s new colleagues over lunch. But then, unexpectedly, JM hears a song in Spanish sung by a busker. This triggers a memory of a family lunch in Spain and he is briefly miles away. He apologises first, and in error in Spanish, a language his boss doesn’t speak. Over lunch, his boss discloses a similar busker experience.

His telling, reported in Table 1, is divided into three episodes: pre-busker, busker and post-busker. For purposes of later comment, Table 1 subdivides the busker episode into unexpected memory (A) and apology (B).

3.2 JM’s path diagrams and temporal experience traces

Figure 2 depicts JM’s path diagrams for each of the three episodes. The path diagrams for the busker episode (Figure 2B) and post-busker episode (Figure 2C) are changes to the first path diagram that corresponds to the pre-busker episode (Figure 2A). There are therefore common nodes and paths across all three episodes but also ones that differentiate his phenomenal states.

Figure 2:

JM’s path diagrams for the pre-busker episode (A), the busker episode (B) and the post-busker episode (C). A link with an arrow head at the end indicates that one node in the path diagram has a positive effect on the state of another. By contrast a link with a filled circle at the end indicates that a node has a negative effect on the state of the node it connects to.

I start with Figure 2A. There are two key factors that express JM’s cognitive-affective nexus (excitement and worry) with a separate node for each. The boss (node B) and the node for colleagues (node C) are simultaneously a source of excitement and worry as indicated by the paths connecting their nodes to the excitement node and to the worry node. In addition, the cityscape (node CS) is a source of excitement (it has a connecting path to the excitement node) and is amplified by the talk of his boss (node B) indicated by the path connecting node B to the one for the cityscape (node CS).

Figure 2B captures the alteration in the field as JM tells of unexpectedly hearing the busker’s song at one point on their route through the city. It triggers a memory (node M). This memory is depicted as a source of excitement and worry. Its connection to the excitement node is indirect via the node for the city scape (CS). The link to the worry node by contrast is direct.

His path diagram (Figure 2C) for the telling of the post-busker episode connects node B (for the boss) and the disclosure (node D). It further connects node D and the worry node indicating that the disclosure reduced worry.

These three path diagrams effectively “arrest” the flow of JM’s subjective experience at critical episodes. Comparing across them captures the rise and fall in excitement and worry and the bases of such changes. For continuous indices of the phenomenal flow JM experienced during his telling of his story, JM also traced the intensity of his experience on a number of dimensions.

Three dimensions seem consistent with JM’s path diagrams. JM is new to his job and hasn’t met his boss before and worries about the impression he makes (one dimension is ‘worry’). JM likes to get on with colleagues and so ‘interpersonal warmth’ is important to him and a second dimension. Lastly, in a new place and in a new job how alive does he feel – how vivid is his experience? ‘vividness’ is the third dimension.

Figure 3 depicts JM’s raw temporal experience traces on these three dimensions: the vertical axis of the graph is felt intensity (e.g., ranging from 0.0 to 1.0) and the horizontal axis is the duration of the talk. Also indicated are the narrative episodes reflected in his talk. We see that vividness peaks at the moment of retelling the involuntary recollection triggered by the Spanish song. A concern with impression peaks shortly afterwards at the apology. (A slip of the pen appears to have disrupted the downward falling vividness trace at this point). On disclosure of his boss’s experience to those around the table at lunch the trace for interpersonal warmth increases dramatically.

Figure 3:

JM’s raw temporal experience traces for his story telling.

The focus of this paper is on the neurophenomenal space but it’s worth noting that data from the two elicitation methods can be related to behavioural data alone. In terms of JM’s speech, we can ask how the ratings of vividness and impression management relate to changes in his speech rate (and pausing) at the retelling of the involuntary memory and the subsequent apology. We know from the trace data and from the path diagram that the involuntary memory was personally significant. And, given the memory was tied to Spanish rather than English, do we find traces of that in his speech. For instance, is there a change in intonation pattern or voice onset times? Returning now to the focus of this paper Section 3.3 considers the value of path diagram and trace data in inferring the mental processes of the cognitive-affective nexus, before envisaging the mapping to the neural bases in Section 3.4.

3.3 Relating and amplifying data of the two methods to infer mental processes underlying the cognitive-affective nexus

Might these two types of phenomenal data work in complementary fashion to help us conjecture the mental processes involved as JM tells his story? One reason why they might be complementary is that temporal experience tracing offers a way to capture the continuity of subjective experience on relevant dimensions but doesn’t capture the interpretation of experience that underlies those tracings. The path diagram offers an interpretation of these in terms of the main forces at play such as the factors of excitement and worry about making a good impression. I consider how these two methods can work together and what further amplification is needed if we are to conjecture the processes at play. As discussed earlier, I presume that narrating the experience recruits sustained attention and that the telling of the experience involves a reconstructive mental simulation of it.

3.4 The neurophenomenal space: relating the attentional, attributional and memorial processes of subjective experience to their neural correlates

For purposes of illustration, I concentrate on the busker episode. During the telling of a story, and so this episode too, the processes comprising subjective experience are expressed through the network of distributed regions of the language network. In telling his story JM used English. For bilingual speakers, such as JM, we know that both languages are active even when only one language is in use (Kroll et al. 2015). Dual activation imposes increased demands on regions involved in speech production compared to those imposed in speakers of just one language (Parker-Jones et al. 2012) with the correlating need to control interference (e.g., Wu and Thierry 2013). What is varying over the various episodes is the relative impact of the attentional, attributional and memorial processes on the language network and so it is their impact that I consider. I note though that in the busker episode JM reports two moments involving a change in language. Whether storytelling retains evidence of the attentional transitions is an open question. If it does, we can make relevant predictions. First, JM’s hearing of the song induced a switch from external attention to an internally-focussed state. The song acted as a salient cue shifting his attention inward. A specific circuit is known to be involved in detecting such cues (e.g., Green and Abutalebi 2013 for details in the context of language use). This dramatic inward shift of attention might also be revealed in the EEG data during the narration (e.g., with a momentary increase in alpha oscillation over parietal regions, Higgins et al. 2021). Second, JM needed to switch from Spanish to English in his apology. His story telling at this point might reveal increased activation in circuits involved in language control (e.g., Abutalebi and Green 2007) and correlating perhaps with the changes in his speech.

Setting these moments on one side, in line with the assumption that a network involved in sustaining attention to story-telling (cf. Rosenberg et al. 2016) recruits a network to simulate the unfolding action, Song et al. (2021) report linkage of these two networks during the comprehension of narratives. Given the reliving assumption, narrating an episode involves a continuous process of mental simulation. Such simulation recruits a wide-scale brain network termed the default mode network (Raichle 2015). However, its activation is likely to be dramatically enhanced as JM mentions an autobiographical memory that is both emotional and salient for him. Cabeza and St. Jacques (2007) report that recalling such a memory increases activation of this network. Interestingly, one region in particular of the default mode network (the medial prefrontal cortex) appears to mediate both autobiographical and mentalizing processes (e.g., Spreng and Grady 2009). The region is activated in mentalizing about oneself and others (see Tan et al. 2022 for electrocorticographic evidence) and, in the emotional assessment of self and other in the context of social interaction (Henry et al. 2021). A reasonable prediction is that this region in particular will show increased activation relative to its activation during the telling of the pre-busker episode.

Wider neural correlates are to be expected. JM recalls the memory in Spanish, the language of the experience, and such a match is known to enhance the richness and vividness in the content of a memory (Marian and Kaushanskaya 2004). Indeed, JM’s unexpected memory is associated with an increase in the rated intensity of vividness and excitement in the path diagram. More emotional personal memories are associated with correlated changes in regions mediating affect, memory and visual imagery (see Cabeza and St. Jacques 2007). We can predict this pattern of correlated changes in the neuroimaging data for JM.

A further prediction speaks directly to embodiment. The unexpected memory occurred as JM and his boss turn a corner. On the assumption that narration involves a simulated retracing of steps then this memory should be tied to a region that tracks entry into new streets. The hippocampus is critically implicated in memory for space and route finding and interestingly, Javadi et al. (2017) found that a right hippocampal region showed increased activity as a new street is entered. Given the assumption that story telling is mental simulation, the prediction is increased activation in this region as JM narrates the busker episode.

In this section I hope to have illustrated how to map subjective experience during storytelling to its neural bases. This mapping is the neurophenomenal space.