Audiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions

doi:10.1016/j.neuropsychologia.2020.107396

Neuropsychologia

Volume 140, 16 March 2020, 107396

https://doi.org/10.1016/j.neuropsychologia.2020.107396 Get rights and content

Highlights

•
Asynchronous and synchronous audiovisual perception are correlated with different brain networks.
•
The width of the TBW is modulated by both top-down and bottom-up factors.
•
The width of the TBW follows a U-shaped developmental trajectory across the lifespan.
•
Atypical TBW is a shared characteristic of different neurodevelopmental disorders.
•
Perceptual and musical training may be able to narrow the audiovisual TBW.

Abstract

To integrate auditory and visual signals into a unified percept, the paired stimuli must co-occur within a limited time window known as the Temporal Binding Window (TBW). The width of the TBW, a proxy of audiovisual temporal integration ability, has been found to be correlated with higher-order cognitive and social functions. A comprehensive review of studies investigating audiovisual TBW reveals several findings: (1) a wide range of top-down processes and bottom-up features can modulate the width of the TBW, facilitating adaptation to the changing and multisensory external environment; (2) a large-scale brain network works in coordination to ensure successful detection of audiovisual (a)synchrony; (3) developmentally, audiovisual TBW follows a U-shaped pattern across the lifespan, with a protracted developmental course into late adolescence and rebounding in size again in late life; (4) an enlarged TBW is characteristic of a number of neurodevelopmental disorders; and (5) the TBW is highly flexible via perceptual and musical training. Interventions targeting the TBW may be able to improve multisensory function and ameliorate social communicative symptoms in clinical populations.

Section snippets

Literature search

A comprehensive search was conducted using the following search terms in PubMed, Elsevier and PsycInfo databases: (“temporal binding” OR “temporal window” OR “time window” OR “temporal acuity” OR “temporal processing” OR “temporal integration” OR “temporal order” OR “simultaneity” OR “*synchrony”) AND (“multisensory” OR “crossmodal” OR “audiovisual” OR “sensory integration”). We only included studies in English involving human subjects. The final search was completed in November 26, 2019 for

Paradigms to measure audiovisual temporal integration

To measure the width of the TBW, a variety of tasks could be used. Table 1 shows all the relevant paradigms, including some specially designed for infants. These tasks could also be classified into explicit (e.g., explicit judgement of simultaneity) and implicit (e.g., implicit fusion of incongruent audiovisual speech stimuli, McGurk fusion) ones. It is unclear whether these tasks capture the same process of multisensory temporal integration. For instance, both the Simultaneity Judgement (SJ)

Factors affecting audiovisual TBW

Both top-down processes, such as attention and expectations, and bottom-up features, like stimulus types and intensity, modulate the likelihood of whether unisensory stimuli would be combined. The following section reviews relevant studies investigating factors affecting the width of the TBW.

Findings from fMRI/PET studies

Detecting audiovisual temporal correlation requires coordinated work of a large-scale brain network, including the auditory (van Atteveldt et al., 2007a, 2007b) and visual (Macaluso et al., 2004) cortices, the fronto-parietal dorsal attention network (Binder, 2015) and some regions crucial for multisensory integration, such as the superior colliculi (Calvert et al., 2001), the insula (Bushara et al., 2001; Lamichhane et al., 2016), the inferior parietal cortex (Adhikari et al., 2013; Dhamala et

Audiovisual TBW across the lifespan

The final product of multisensory integration is shaped by the relative reliance on low-level stimulus characteristics including temporal relationships and experienced-related learned associations (see review, Murray et al., 2016). Developmentally, there is a shift from heavy dependence on stimulus characteristics in early life to increasing emphasis on learned associations as one progresses into adulthood (Murray et al., 2016). Specifically for the ability of utilizing temporal cues to

Neurodevelopmental disorders

A newly published review article summarized literature on a broader scope of multisensory processing not restricted to audiovisual domain and temporal factors (Wallace et al., 2020). They have found altered multisensory function in several neurodevelopmental and neuropsychiatric conditions and such atypical multisensory integration has been shown to correlate with higher-order cognitive and social abilities (Wallace et al., 2020).

Specifically for the temporal aspect of multisensory processing,

Modifiability of the audiovisual TBW

Multisensory temporal integration has a high degree of plasticity even in adult brains. On the one hand, the width of the TBW can be significantly narrowed with perceptual training (Powers et al., 2009). On the other hand, the PSS can be shifted towards the auditory or visual-leading side after transient or intensive exposure to temporally-asynchronous pairs (Fujisaki et al., 2004; Van der Burg and Goodbourn, 2015), which is a process called temporal recalibration. As this review has an

Conclusions

This paper systematically reviews studies investigating audiovisual TBW. We have summarized the neural mechanisms of different cognitive stages underlying audiovisual temporal integration, indicating a widely distributed brain network responsible for such a basic multisensory function. The width of the TBW follows a U-shaped pattern across the lifespan, which does not mature until late adolescence and rebound in size again in old age. An enlarged TBW, reflecting impaired multisensory

Declaration of competing interest

No conflict of interest to be declared.

Acknowledgments

This study was supported by a grant from the National Science Foundation China (31970997), Beijing Training Project for Leading Talents in S&T (Z151100000315020), and the CAS Key Laboratory of Mental Health, Institute of Psychology.

References (276)

L.E. Bahrick et al.
Intersensory redundancy promotes infant detection of prosody in infant-directed speech
J. Exp. Child Psychol.
(2019)
S.H. Baum et al.
Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder
Prog. Neurobiol.
(2015)
S. Beker et al.
Ripe for solution: delayed development of multisensory processing in autism and its remediation
Neurosci. Biobehav. Rev.
(2018)
P. Bertelson et al.
Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment
Int. J. Psychophysiol.
(2003)
E. Biau et al.
Hand gestures as visual prosody: BOLD responses to audio-visual alignment are modulated by the communicative nature of the stimuli
Neuroimage
(2016)
G.M. Bidelman et al.
Neural correlates of enhanced audiovisual processing in the bilingual brain
Neuroscience
(2019)
M. Binder
Neural correlates of audiovisual temporal processing-comparison of temporal order and simultaneity judgments
Neuroscience
(2015)
L. Blomert et al.
Multi-sensory learning and learning to read
Int. J. Psychophysiol.
(2010)
G.A. Calvert et al.
Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect
Neuroimage
(2001)
R. Cecere et al.
Individual differences in alpha frequency drive crossmodal illusory perception
Curr. Biol.
(2015)

Y.M. Chan et al.

Spatio-temporal patterns of event-related potentials related to audiovisual synchrony judgments in older adults

Neurobiol. Aging

(2017)

L. Chen et al.

Crossmodal synesthetic congruency improves visual timing in dyslexic children

Res. Dev. Disabil.

(2016)

Y.C. Chen et al.

The development of the perception of audiovisual simultaneity

J. Exp. Child Psychol.

(2016)

Y.C. Chen et al.

Developmental changes in the perception of visuotactile simultaneity

J. Exp. Chid Psychol.

(2018)

E. Ciaramelli et al.

Top-down and bottom-up attention to memory: a hypothesis (AtoM) on the role of the posterior parietal cortex in memory retrieval

Neuropsychologia

(2008)

L.M. Curtindale et al.

Effects of multimodal synchrony on infant attention and heart rate during events with social and nonsocial stimuli

J. Exp. Child Psychol.

(2019)

L. de Boer-Schellekens et al.

No evidence for impaired multisensory integration of low-level audiovisual stimuli in adolescents and young adults with autism spectrum disorders

Neuropsychologia

(2013)

L. De Boer-Schellekens et al.

Sound improves diminished visual temporal sensitivity in schizophrenia

Acta Psychol. (Amst).

(2014)

L. De Boer-Schellekens et al.

Sound improves diminished visual temporal sensitivity in schizophrenia

Acta Psychol. (Amst).

(2014)

M.A. De Niear et al.

Perceptual training enhances temporal acuity for multisensory speech

Neurobiol. Learn. Mem.

(2018)

X. Delbeuck et al.

Is Alzheimer's disease a disconnection syndrome?. Evidence from a crossmodal audio-visual illusory experiment

Neuropsychologia

(2007)

A. Desantis et al.

Action-outcome learning and prediction shape the window of simultaneity of audiovisual outcomes

Cognition

(2016)

M. Dhamala et al.

Multisensory integration for timing engages different brain networks

Neuroimage

(2007)

A. Diederich et al.

Assessing age-related multisensory enhancement with the time-window-of-integration model

Neuropsychologia

(2008)

T. Falck-Ytter et al.

Human infants orient to biological motion rather than audiovisual synchrony

Neuropsychologia

(2011)

J.I. Feldman et al.

Audiovisual multisensory integration in individuals with autism spectrum disorder: a systematic review and meta-analysis

Neurosci. Biobehav. Rev.

(2018)

F. Ferri et al.

Higher proneness to multisensory illusions is driven by reduced temporal sensitivity in people with high schizotypal traits. Conscious

Cognition

(2018)

J.R. Foucher et al.

Low time resolution in schizophrenia. Lengthened windows of simultaneity for visual, auditory and bimodal stimuli

Schizophr. Res.

(2007)

R. Franciotti et al.

Neuromagnetic responses reveal the cortical timing of audiovisual synchrony

Neuroscience

(2011)

E.D. Freeman et al.

Sight and sound out of synch: fragmentation and renormalisation of audiovisual integration and subjective timing

Cortex

(2013)

D.C. Glahn et al.

Meta-analysis of gray matter anomalies in schizophrenia: application of anatomic likelihood estimation and network analysis

Biol. Psychiatr.

(2008)

N. Hahn et al.

Impairments of multisensory integration and cross-sensory learning as pathways to dyslexia

Neurosci. Biobehav. Rev.

(2014)

E.E. Hannon et al.

Babies know bad dancing when they see it: older but not younger infants discriminate between synchronous and asynchronous audiovisual musical displays

J. Exp. Child Psychol.

(2017)

A. Hillock-Dunn et al.

The temporal binding window for audiovisual speech: children are like little adults

Neuropsychologia

(2016)

A.R. Hillock et al.

Binding of sights and sounds: age-related changes in multisensory temporal processing

Neuropsychologia

(2011)

B.M. Adhikari et al.

Temporal-order judgment of audiovisual events involves network activity between parietal and prefrontal cortices

Brain Connect.

(2013)

D. Alais et al.

Multisensory perceptual learning of temporal order: audiovisual learning transfers to vision but not audition

PloS One

(2010)

M.J. Allman et al.

Pathophysiological distortions in time perception and timed performance

Brain

(2012)

B.E. Arikan et al.

Voluntary and involuntary movements widen the window of subjective simultaneity

Iperception

(2017)

L.E. Bahrick

Intermodal learning in infancy: learning on the basis of two kinds of invariant relations in audible and visible events

Child Dev.

(1988)

L.E. Bahrick et al.

Intersensory redundancy facilitates discrimination of tempo in 3-month-old infants

Dev. Psychobiol.

(2002)

L.E. Bahrick et al.

Intersensory redundancy guides attentional selectivity and perceptual learning in infancy

Dev. Psychol.

(2000)

D.J.K. Barrett et al.

Evidence for multisensory integration in the elicitation of prior entry by bimodal cues

Exp. Brain Res.

(2012)

A. Basharat et al.

Simultaneity and temporal order judgments are coded differently and change with age: an event-related potential study

Front. Integr. Neurosci.

(2018)

O. Baumann et al.

Neural correlates of temporal complexity and synchrony during audiovisual correspondence detection

eNeuro

(2018)

J.M. Bebko et al.

Discrimination of temporal synchrony in intermodal events by children with autism and children with developmental disabilities without autism

J. Child Psychol. Psychiatry Allied Discip.

(2006)

G. Bedard et al.

Impaired timing of audiovisual events in the elderly

Exp. Brain Res.

(2016)

J. Bhat et al.

Putative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony

J. Neurophysiol.

(2015)

G.M. Bidelman

Musicians have enhanced audiovisual multisensory binding: experience-dependent effects in the double-flash illusion

Exp. Brain Res.

(2016)

G.M. Bidelman et al.

Enhanced temporal binding of audiovisual information in the bilingual brain

Biling. Lang. Cognit.

(2019)

Cited by (38)

Audiovisual spatial ventriloquism is reduced in musicians
2023, Hearing Research
There is great scientific and public interest in claims that musical training improves general cognitive and perceptual abilities. While this is controversial, recent and rather convincing evidence suggests that musical training refines the temporal integration of auditory and visual stimuli at a general level. We investigated whether musical training also affects integration in the spatial domain, via an auditory localisation experiment that measured ventriloquism (where localisation is biased towards visual stimuli on audiovisual trials) and recalibration (a unimodal localisation aftereffect). While musicians (n = 22) and non-musicians (n = 22) did not have significantly different unimodal precision or accuracy, musicians were significantly less susceptible than non-musicians to ventriloquism, with large effect sizes. We replicated these results in another experiment with an independent sample of 24 musicians and 21 non-musicians. Across both experiments, spatial recalibration did not significantly differ between the groups even though musicians resisted ventriloquism. Our results suggest that the multisensory expertise afforded by musical training refines spatial integration, a process that underpins multisensory perception.
Preterm birth and neonatal white matter microstructure in in-vivo reconstructed fiber tracts among audiovisual integration brain regions
2023, Developmental Cognitive Neuroscience
Individuals born preterm are at risk of developing a variety of sequelae. Audiovisual integration (AVI) has received little attention despite its facilitating role in the development of socio-cognitive abilities. The present study assessed the association between prematurity and in-vivo reconstructed fiber bundles among brain regions relevant for AVI. We retrieved data from 63 preterm neonates enrolled in the Developing Human Connectome Project (http://www.developingconnectome.org/) and matched them with 63 term-born neonates from the same study by means of propensity score matching. We performed probabilistic tractography, DTI and NODDI analysis on the traced fibers. We found that specific DTI and NODDI metrics are significantly associated with prematurity in neonates matched for postmenstrual age at scan. We investigated the spatial overlap and developmental order of the reconstructed tractograms between preterm and full-term neonates. Permutation-based analysis revealed significant differences in dice similarity coefficients and developmental order between preterm and full term neonates at the group level. Contrarily, no group differences in the amount of interindividual variability of DTI and NODDI metrics were observed. We conclude that microstructural detriment in the reconstructed fiber bundles along with developmental and morphological differences are likely to contribute to disadvantages in AVI in preterm individuals.
The influence of temporal asynchrony on character-speech integration in Chinese children with and without dyslexia: An ERP study
2022, Brain and Language
Citation Excerpt :
However, both the TOJ and SJ tasks were designed to explore the temporal acuity across individuals but not to reflect cross-modal integration per se. Therefore, the hypothesis that reading deficit may be a result of abnormally widened TBWs warrants further investigation (Zhou et al., 2020). In a series of ERP studies, mismatch negativity (MMN) was used to investigate the influence of SOAs on LSI (Froyen et al., 2008; Froyen et al., 2009; Froyen et al., 2011).
Dyslexic readers have been reported to show abnormal temporal acuity and multisensory integration deficiency. Here, we investigated the influence of temporal intervals on Chinese character-speech integration in children with and without dyslexia. Visual characters were presented synchronously to the onset of speech sounds (AV0) or before speech sound by 300 ms (AV300). Event-related potentials (ERP) evoked by congruent condition (speech sounds presented with congruent Chinese characters) and by baseline condition (speech sounds presented with Korean characters) were compared. Typically developing (TD) children exhibited congruency effect in AV0 condition, whereas dyslexic children exhibited congruency effect in AV300 condition. Moreover, congruency effect in TD children was due to enhanced neural activation to congruent trials, congruency effect in dyslexic children was contributed by neural suppression for baseline trials. These results suggested that different underlying mechanisms were involved in character-speech integration for typical and dyslexic children.
Fiber tracing and microstructural characterization among audiovisual integration brain regions in neonates compared with young adults
2022, NeuroImage
Citation Excerpt :
To our knowledge, this is the first study to directly assess such connections and to characterize their properties around term age. Whereas AVI has been extensively studied in adults (Zhou et al., 2020) and more recently in young infants from a functional point of view (e.g., Altvater-Mackensen and Grossmann, 2016; Bristow et al., 2009; Kushnerenko et al., 2008), little is known about the structural pathways of the neonatal brain supporting AVI. These results contribute to the literature on AVI development and to the discussion on its ontogeny.
Audiovisual integration has been related with cognitive-processing and behavioral advantages, as well as with various socio-cognitive disorders. While some studies have identified brain regions instantiating this ability shortly after birth, little is known about the structural pathways connecting them. The goal of the present study was to reconstruct fiber tracts linking AVI regions in the newborn in-vivo brain and assess their adult-likeness by comparing them with analogous fiber tracts of young adults. We performed probabilistic tractography and compared connective probabilities between a sample of term-born neonates (N = 311; the Developing Human Connectome Project (dHCP, http://www.developingconnectome.org) and young adults (N = 311 The Human Connectome Project; https://www.humanconnectome.org/) by means of a classification algorithm. Furthermore, we computed Dice coefficients to assess between-group spatial similarity of the reconstructed fibers and used diffusion metrics to characterize neonates’ AVI brain network in terms of microstructural properties, interhemispheric differences and the association with perinatal covariates and biological sex. Overall, our results indicate that the AVI fiber bundles were successfully reconstructed in a vast majority of neonates, similarly to adults. Connective probability distributional similarities and spatial overlaps of AVI fibers between the two groups differed across the reconstructed fibers. There was a rank-order correspondence of the fibers’ connective strengths across the groups. Additionally, the study revealed patterns of diffusion metrics in line with early white matter developmental trajectories and a developmental advantage for females. Altogether, these findings deliver evidence of meaningful structural connections among AVI regions in the newborn in-vivo brain.
Anterior cingulate glutamate levels associate with functional activation and connectivity during sensory integration in schizophrenia: a multimodal <sup>1</sup>H-MRS and fMRI study
2023, Psychological Medicine
Audio–Visual Fission Illusion and Individual Alpha Frequency: Perspective on Buergers and Noppeney (2022)
2024, Journal of Cognitive Neuroscience

View all citing articles on Scopus

View full text

Review articleAudiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions

Highlights

Abstract

Section snippets

Literature search

Paradigms to measure audiovisual temporal integration

Factors affecting audiovisual TBW

Findings from fMRI/PET studies

Audiovisual TBW across the lifespan

Neurodevelopmental disorders

Modifiability of the audiovisual TBW

Conclusions

Declaration of competing interest

Acknowledgments

J. Exp. Child Psychol.

Prog. Neurobiol.

Neurosci. Biobehav. Rev.

Int. J. Psychophysiol.

Neuroimage

Neuroscience

Neuroscience

Int. J. Psychophysiol.

Neuroimage

Curr. Biol.

Neurobiol. Aging

Res. Dev. Disabil.

J. Exp. Child Psychol.

J. Exp. Chid Psychol.

Neuropsychologia

J. Exp. Child Psychol.

Neuropsychologia

Acta Psychol. (Amst).

Acta Psychol. (Amst).

Neurobiol. Learn. Mem.

Neuropsychologia

Cognition

Neuroimage

Neuropsychologia

Neuropsychologia

Neurosci. Biobehav. Rev.

Cognition

Schizophr. Res.

Neuroscience

Cortex

Biol. Psychiatr.

Neurosci. Biobehav. Rev.

J. Exp. Child Psychol.

Neuropsychologia

Neuropsychologia

Temporal-order judgment of audiovisual events involves network activity between parietal and prefrontal cortices

Brain Connect.

Multisensory perceptual learning of temporal order: audiovisual learning transfers to vision but not audition

PloS One

Pathophysiological distortions in time perception and timed performance

Brain

Voluntary and involuntary movements widen the window of subjective simultaneity

Iperception

Intermodal learning in infancy: learning on the basis of two kinds of invariant relations in audible and visible events

Child Dev.

Intersensory redundancy facilitates discrimination of tempo in 3-month-old infants

Dev. Psychobiol.

Intersensory redundancy guides attentional selectivity and perceptual learning in infancy

Dev. Psychol.

Evidence for multisensory integration in the elicitation of prior entry by bimodal cues

Exp. Brain Res.

Simultaneity and temporal order judgments are coded differently and change with age: an event-related potential study

Front. Integr. Neurosci.

Neural correlates of temporal complexity and synchrony during audiovisual correspondence detection

eNeuro

Discrimination of temporal synchrony in intermodal events by children with autism and children with developmental disabilities without autism

J. Child Psychol. Psychiatry Allied Discip.

Impaired timing of audiovisual events in the elderly

Exp. Brain Res.

Putative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony

J. Neurophysiol.

Musicians have enhanced audiovisual multisensory binding: experience-dependent effects in the double-flash illusion

Exp. Brain Res.

Enhanced temporal binding of audiovisual information in the bilingual brain

Biling. Lang. Cognit.

Review article
Audiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions