Elsevier

Cortex

Volume 134, January 2021, Pages 223-238
Cortex

Special Issue “Multisensory integration”: Research Report
Exposure to first-person shooter videogames is associated with multisensory temporal precision and migraine incidence

https://doi.org/10.1016/j.cortex.2020.10.009Get rights and content

Abstract

Adaptive interactions with the environment require optimal integration and segregation of sensory information. Yet, temporal misalignments in the presentation of visual and auditory stimuli may generate illusory phenomena such as the sound-induced flash illusion, in which a single flash paired with multiple auditory stimuli induces the perception of multiple illusory flashes. This phenomenon has been shown to be robust and resistant to feedback training. According to a Bayesian account, this is due to a statistically optimal combination of the signals operated by the nervous system. From this perspective, individual susceptibility to the illusion might be moulded through prolonged experience. For example, repeated exposure to the illusion and prolonged training sessions partially impact on the reported illusion.

Therefore, extensive and immersive audio-visual experience, such as first-person shooter videogames, should sharpen individual capacity to correctly integrate multisensory information over time, leading to more veridical perception. We tested this hypothesis by comparing the temporal profile of the sound-induced illusion in a group of expert first-person shooter gamers and a non-players group. In line with the hypotheses, gamers experience significantly narrower windows of illusion (~87 ms) relative to non-players (~105 ms), leading to higher veridical reports in gamers (~68%) relative to non-players (~59%). Moreover, according to recent literature, we tested whether audio-visual intensive training in gamers could be related to the incidence of migraine, and found that its severity may be directly proportioned to the time spent on videogames. Overall, these results suggest that continued training within audio-visual environments such as first-person shooter videogames improves temporal discrimination and sensory integration. This finding may pave the way for future therapeutic strategies based on self-administered multisensory training. On the other hand, the impact of intensive training on visual-related stress disorders, such as migraine incidence, should be taken into account as a risk factor during therapeutic planning.

Introduction

Our perception and interactions with the external world do not consist of isolated sensory events, but rather a rich combination of multisensory experiences. This integration of information across different sensory modalities represents a ubiquitous operation in everyday human behaviors that shapes our perception (Sutherland et al., 2014), improving our ability to interact with the environment adaptively, allowing for enhanced detection (Leo and Noppeney, 2014; Leo et al., 2011; Lovelace et al., 2003; Ramos-Estebanez et al., 2007), more accurate localization (Leo and Noppeney, 2014; Nelson et al., 1998; Wilkinson et al., 1996), and faster reactions (Diederich and Colonius, 2004; Hershenson, 1962;Romei et al., 2007). In addition to these highly adaptive benefits, multisensory integration can also be seen in a host of perceptual illusions. One of the more compelling illusions is the ‘sound-induced flash illusion’ (SIFI) (Hirst et al., 2020; Keil, 2020; Shams et al., 2000, 2002), that occurs when a single flash is presented along with two or more beeps and observers often report seeing two or more flashes, demonstrating that visual perception can be radically altered by signals of other modalities. Importantly, studies have demonstrated that these interactions between the senses are highly dependent upon the temporal characteristics of the stimuli that are combined, such as their level of synchrony. Indeed, it has been shown that multisensory stimuli presented in close temporal proximity are often integrated into a single, unified percept (Andersen et al., 2004; Bastiaansen et al., 2020; Migliorati et al., 2019; Shams et al., 2000; Stein and Meredith, 1993). This perceptual binding over a given temporal interval is best captured in the construct of a multisensory temporal binding window (TBW); (Colonius and Diederich, 2004; Hairston et al., 2005; Powers et al., 2009). Within this window, the combination of information between two different modalities across a range of stimulus asynchronies results in significant alterations at the neural, behavioral and perceptual responses. In line with this idea is the observation that the closer the temporal proximity between two presented stimuli, the more likely they will be integrated into a single percept. On the other hand, as the temporal interval between the stimuli (i.e., the stimulus onset asynchrony, SOA) increases, the likelihood of multisensory interactions decreases (Chan et al., 2018; Shams et al., 2000; Spence et al., 2003; Vroomen and Keetels, 2010). Interestingly, it has been suggested that the brain uses a mechanism similar to Bayesian inference (Knill and Pouget, 2004; Shams et al., 2005) to decide whether, to what degree, and how (in which direction) to integrate the signals from auditory and visual modalities, and that the SIFI can be considered as an epiphenomenon of a statistically optimal computational strategy. In particular, auditory and visual stimuli appear to be integrated into a single perceptual representation at SOAs from physical simultaneity (i.e., 0 msec apart) up to ~100–150 msec, after which the two stimuli are perceived as distinct (Schneider and Bavelier, 2003; Zampini et al., 2005). Accordingly, this temporal window has been viewed as reflecting the typical temporal window of multisensory integration. However, consistent differences across healthy individuals have been observed (Donohue et al., 2015; Hernández et al., 2019; Stevenson et al., 2012; Setti et al., 2011; Stone et al., 2001) and altered multisensory integration with both reduced and wider audio-visual TBW has been documented in atypical populations such as in individuals with autistic spectrum disorders (Stevenson et al., 2014; van Laarhoven et al., 2019; Wallace and Stevenson, 2014; Yaguchi and Hidaka, 2018; Zhou et al., 2018), dyslexia (Bastien-Toniazzo et al., 2010; Hairston et al., 2005) and schizophrenia (De Gelder et al., 2003, 2005; De Jong et al., 2009; Foucher et al., 2007; Pearl et al., 2009; Ross et al., 2007; Szycik et al., 2009).

In line with a Bayesian account, it has been observed that experience can narrow the TBW, for example, a perceptual training (<5 days) in either a combined or unisensory regimen (Powers et al., 2009; Setti et al., 2014; Stevenson et al., 2013; but see; Powers et al., 2016). However, multisensory training might affect the sensitivity to SIFI differently depending on age and task. Indeed, multisensory abilities have been shown to undergo developmental changes (Hirst et al., 2019), as in older adults training seems to be more effective (Setti et al., 2014) relative to younger ones (Powers et al., 2016). Moreover the training task properties possibly impact the outcome of perceptual learning through different transfer mechanisms (McGovern et al., 2016a, b). On the other hand, certain long-term perceptual–cognitive experiences such as musical (Bidelman, 2016) and bilingual (Bidelman & Heath, 2019) exposure, prove the influence of expertise in shaping multisensory integration profiles.

Similarly to what has been found for musicians (Bidelman, 2016) or bilinguals (Bidelman & Heath, 2019), the current study investigates a novel, not yet tested hypothesis (for recent reviews see Hirst et al., 2020; Keil et al., 2020) that an intensive audiovisual experience, such as videogame playing, can sharpen audio-visual processing and their temporal binding window. Moreover, we tested a related hypothesis that together with positive performance effects, intensive videogame playing may be related to an increased incidence of migraine, a neurological condition determined by altered visual cortex excitability and reduced visual contrast sensitivity (Asher et al., 2018; Aurora et al., 1998, Aurora et al., 2003; Brigo et al., 2013; O'Hare and Hibbard, 2016).

Videogames have an astonishing effect on the player's visual system. In several studies, action videogame players have outperformed the non-players in visual information processing abilities (Riesenhuber, 2004), including selective attention (Belchior et al., 2013; Dye et al., 2009; Green and Bavelier, 2003), visual search efficiency (Castel et al., 2005), visuospatial attention (Clark et al., 2011; Spence and Feng, 2010; West et al., 2008), contrast sensitivity (Li et al., 2011) and visual interference suppression (Hazarika et al., 2018). In particular, action videogame players used to play first-person shooter game (FPSG) which is an action videogame involving first person's perspective in which the player experiences the gaming environment through the eyes of the protagonist. The player controls a character or an avatar of the game which performs multiple tasks at the same time, and this has been found to improve players' ability to simultaneously track multiple moving visual items (Green & Bavelier, 2006), spatial abilities (Quaiser-Pohl, Geiser & Lehmann, 2006) and divided attention abilities (Greenfield, DeWinstanley, Kilpatrick & Kaye, 1994). Player needs to react through mouse, keyboard or joystick resulting in faster hand-eye coordination and quick reflective actions (Griffith et al., 1983). Importantly, videogames are inherently multisensory, with first-person shooter and other action games often having both auditory and visual cues that are relevant to an appropriate behavioral response, also considering that the presence of tactile stimulation via the haptic device (e.g., joystick) offers an additional sensory source to be implemented in some cases (Archambault et al., 2007). In line with the multisensory nature of the game, it has been found that videogame players have a more fine-tuned sense of temporal synchrony that enables a greater ability related to the non-players to notice slight asynchronies between auditory and visual stimuli (Donohue et al., 2015), supporting the notion that temporal integration can be manipulated by prior experiences (Harrar and Harris, 2008; Vroomen et al., 2004).

The aim of this study was twofold. First, we aimed to determine whether extensive videogame experience enhances audio-visual processing and narrows the temporal binding window for combining multisensory cues. To test our hypothesis, we submitted the sound-induced flash illusion (SIFI) to first-person shooter game players (FPSG) and non-players (NP) and parametrically varied the onset asynchrony between the first audio-visual pair and the second auditory stimulus. Since it has been demonstrated that individuals with a narrower TBW have an enhanced capacity to distinguish between asynchronous audio-visual inputs (Stevenson et al., 2012), we predict FPSG to show also a reduced temporal window in which they experience the illusion (TWI) which in turn may account for the reduced overall proneness to the illusion (see Ferri, Venskus, Fotia, Cooke & Romei, 2018).

Second, we aimed at testing the relation between videogame experience and migraine based on their link to visual cortical excitability. Noninvasive brain stimulation techniques, such as the tDCS and tACS, have shown to change the SIFI in healthy volunteers, by modulating the excitability of the primary visual cortex (Bolognini, Rossetti, Casati, Mancini & Vallar, 2011) and the temporal window within which the participants perceive the illusion by manipulating the frequency of the occipital alpha activity (Cecere, Rees & Romei, 2015, an oscillatory activity associated with the level of visual cortex excitability within (Romei et al., 2008) and across participants (Romei, Rihs, Brodbeck & Thut, 2008). These findings, which demonstrated that visual cortical excitability has an impact in the SIFI, are also supported by studies on migraine, a condition associated with pathologic cortical hyperexcitability, that results in reduced multisensory illusion (Brighina et al., 2015). In line with this data, migraine is often associated with the excessive use of digital equipment (Torsheim et al., 2010; Xavier et al., 2015). Therefore, we expect that the visual sensory stress due to the intensive and immersive training of the audio-visual integration system of FPSG may lead to higher visual cortex excitability and consequently, to a higher headache-related disability, which may represent a consequence related to the modulation operated at the level of the cortical substrate associated with reduced temporal windows within which gamers experience the SIFI.

Section snippets

Participants

An initial online screening on a wide sample of ~400 responders was performed in order to identify participants with different levels of gaming experience. Sample size, with all inclusion and exclusion criteria was established prior to data analysis and we report all manipulation and measures in the study. Based on a priori power analysis considering previous studies (Donohue et al., 2015; Ferri et al., 2018; Hazarika et al., 2018) and returning an estimated sample size of 42 participants

Temporal Window of Illusion (TWI)

Participants' responses from the behavioural task were used to calculate the temporal window within which the illusion was maximally perceived. To this end, the percentage of trials where the illusion (i.e., two flashes) was experienced was first plotted as a function of the SOAs.

We fitted data to a psychometric sigmoid function [y = a+b/(1+exp(-(x-c)/d)); a = lower asymptote; b = upper asymptote; c = inflection point; d = slope]. The obtained SOA (in ms) corresponding to the inflection point

Temporal Window of Illusion

In line with a role of experience and priors in shaping perception, our primary working hypothesis would predict a shrinking of the temporal window within which the FPSG experience the illusion relative to the NP group. Accordingly, our results revealed for the first time that video gamer individuals showed a significantly reduced TWI (87.75 ms) relative to non-players individuals (105.32 ms) by about 18 ms [independent sample one-tailed t-test: t(40) = −2.49; p = .008; d = −.77 see Fig. 2a,b].

Discussion

In the present study we asked whether the proneness and temporal sensitivity to the sound-induced flash illusion (SIFI) is clustered as a function of training on first-person shooter videogames (FPSG). To this aim, we tested a group of expert players and contrasted their performance against a group of non-players. The main outcome of the study confirms that FPSG, relative to the NP, show smaller temporal windows within which they experience the illusion and higher veridical reports of

Conclusions

A reduced temporal window of integration, leading to a more veridical perception of visual stimulus quantity, seems to characterize people with extensive experience at first-person videogame shooting. This is possibly due to their daily practice in time-sensitive fast-paced multisensory stimuli. Bayesian mechanisms may be at play for multisensory stimuli to be appropriately integrated or segregated in time to adapt and succeed with the shooter performance functionally. On the other hand,

Author contributions

V.R. and P.DL. developed the study concept; V.R. P.DL. A.T. and S.B. contributed to the study design; P.DL. and S.S. performed testing and data collection; P.DL performed the data analysis; V.R. and S.B. wrote the first draft of the manuscript; all authors contributed to and approved the final version of the manuscript for submission.

Open practices

The study in this article earned Open Materials and Open Data badges for transparent practices. Materials and data for the study are available at https://osf.io/5h76y/.

Declaration of competing interest

The authors declare no conflict of interest.

Acknowledgments

This work was supported by grants from Bial Foundation awarded to V.R. (204/18), and Ministero della Salute, Italy (GR-2018-12365733) awarded to S. B.

References (152)

  • B. De Gelder et al.

    Multisensory integration of emotional faces and voices in schizophrenics

    Schizophrenia Research

    (2005)
  • J.J. De Jong et al.

    Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect

    Schizophrenia Research

    (2009)
  • M.W.G. Dye et al.

    The development of attention skills in action video game players

    Neuropsychologia

    (2009)
  • F. Ferri et al.

    Higher proneness to multisensory illusions is driven by reduced temporal sensitivity in people with high schizotypal traits

    Consciousness and Cognition

    (2018)
  • J.R. Foucher et al.

    Low time resolution in schizophrenia. Lengthened windows of simultaneity for visual, auditory and bimodal stimuli

    Schizophrenia Research

    (2007)
  • S. Franceschini et al.

    Action video games make dyslexic children read better

    Current Biology

    (2013)
  • C.S. Green et al.

    Improved probabilistic inference as a general learning mechanism with action video games

    Current Biology

    (2010)
  • P.M. Greenfield et al.

    Action video games and informal education: Effects on strategies for dividing visual attention

    Journal of Applied Developmental Psychology

    (1994)
  • C.S. Green et al.

    Enumeration versus multiple object tracking: The case of action video game players

    Cognition

    (2006)
  • J. Hazarika et al.

    Neural modulation in action video game players during inhibitory control function: An EEG study using discrete wavelet transform

    Biomedical Signal Processing and Control

    (2018)
  • H.P. Helfrich et al.

    A trust region algorithm for parametric curve and surface fitting

    Journal of Computational and Applied Mathematics

    (1996)
  • S.C. Herholz et al.

    Musical Training as a Framework for Brain Plasticity: Behavior, Function, and Structure

    Neuron

    (2012)
  • R.J. Hirst et al.

    What you see is what you hear: Twenty years of research using the Sound-Induced Flash illusion

    Neuroscience and Biobehavioral Reviews

    (2020)
  • D.C. Knill et al.

    The Bayesian brain: The role of uncertainty in neural coding and computation

    Trends in Neurosciences

    (2004)
  • C.T. Lovelace et al.

    An irrelevant light enhances auditory detection in humans: A psychophysical analysis of multisensory integration in stimulus detection

    Cognitive Brain Research

    (2003)
  • M. Mathers et al.

    Electronic media use and adolescent health and well-being: Cross-sectional community study

    Academic Pediatrics

    (2009)
  • D.P. McGovern et al.

    Task-specific transfer of perceptual learning across sensory modalities

    Current Biology

    (2016)
  • N.A. Merriman et al.

    Successful balance training is associated with improved multisensory function in fall-prone older adults

    Computers in Human Behavior

    (2015)
  • M. Aickin et al.

    Adjusting for multiple testing when reporting research results: The Bonferroni vs Holm methods

    American Journal of Public Health

    (1996)
  • D. Archambault et al.

    Computer games and visually impaired people

    Upgrade

    (2007)
  • J. Asher et al.

    Typical lateral interactions, but increased contrast sensitivity, in migraine-with-Aura

    Vision

    (2018)
  • S.K. Aurora et al.

    Transcranial magnetic stimulation confirms hyperexcitability of occipital cortex in migraine

    Neurology

    (1998)
  • S.K. Aurora et al.

    The threshold for phosphenes is lower in migraine

    Cephalalgia: an International Journal of Headache

    (2003)
  • M. Bastiaansen et al.

    Are alpha oscillations instrumental in multisensory synchrony perception?

    Brain Research

    (2020)
  • Bastien-ToniazzoM. et al.

    Audio-visual perception and integration in developmental dyslexia: An exploratory study using the McGurk effect

    Current Psychology Letters

    (2010)
  • D. Bavelier et al.

    Brain plasticity through the life span: Learning to learn and action video games

    Annual Review of Neuroscience

    (2012)
  • B. Bediou et al.

    Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills

    Psychological Bulletin

    (2018)
  • G.M. Bidelman

    Musicians have enhanced audiovisual multisensory binding: Experience-dependent effects in the double-flash illusion

    Experimental Brain Research

    (2016)
  • G.M. Bidelman et al.

    Enhanced temporal binding of audiovisual information in the bilingual brain

    Bilingualism

    (2019)
  • A.J. Bishara et al.

    Confidence intervals for correlations when data are not normal

    Behavior Research Methods

    (2017)
  • F. Brighina et al.

    Visual cortex hyperexcitability in migraine in response to sound-induced flash illusions

    Neurology

    (2015)
  • F. Brigo et al.

    Primary visual cortex excitability in migraine: a systematic review with meta-analysis

    Neurological Sciences

    (2012)
  • J.S. Chan et al.

    The number of stimulus-onset asynchronies affects the perception of the sound-induced flash illusion in young and older adults

    Multisensory Research

    (2018)
  • J. Chan et al.

    Expanded temporal binding windows in people with mild cognitive impairment

    Current Alzheimer Research

    (2015)
  • W. Chan et al.

    Bootstrap standard error and confidence intervals for the correlation corrected for range restriction: A simulation study

    Psychological Methods

    (2004)
  • G.M. Cicchini et al.

    The functional role of serial dependence

    Proceedings of the Royal Society B: Biological Sciences

    (2018)
  • H. Colonius et al.

    Multisensory interaction in saccadic reaction time: A time-window-of- integration model

    Journal of Cognitive Neuroscience

    (2004)
  • J. Cooke et al.

    Oscillatory properties of functional connections between sensory areas mediate cross-modal illusory perception

    Journal of Neuroscience

    (2019)
  • G. Cosentino et al.

    O047. The sound-induced flash illusions reveal visual cortex hyperexcitability in cluster headache

    Journal of Headache and Pain

    (2015)
  • G. Dale et al.

    Annals of the New York academy of sciences A new look at the cognitive neuroscience of video game play

    Annals of the New York Academy of Sciences

    (2020)
  • Cited by (0)

    View full text