Research ReportVisual mismatch responses index surprise signalling but not expectation suppression
Introduction
We are highly adept at learning recurring patterns and statistical regularities that occur in our sensory environments, and we can exploit this knowledge to discriminate between commonplace and unusual sensory events. The computations that underpin this capability likely rely on statistics learned over time-scales ranging from hundreds of milliseconds to several minutes (Maheu et al., 2019; Ulanovsky et al., 2004). Characterising the computations that occur within different brain regions can help us understand how we generate and evaluate internal models of our environment, and how these models are used to facilitate detection of potential rewards (Schultz, 2016) and enact adaptive adjustments to decision-making strategies (Wessel, 2017; Wessel & Huber, 2019).
Detection of novel or unusual events has been investigated using visual oddball designs in which a stimulus is presented in high and low probability contexts. For example, in the high probability context, a critical stimulus A is presented frequently as the ‘standard’ stimulus, interspersed with a rare ‘deviant’ stimulus B (e.g., AABAAAAAAABAAAB). In the low probability context, the standard stimulus A is instead presented as a rare deviant (e.g., BBBBBABBBBBABBB). Comparisons of visual stimulus-evoked event-related potentials (ERPs) recorded using electroencephalography (EEG) reveal more negative-going waveforms evoked by deviants at posterior scalp electrodes from 150 to 300 msec following stimulus onset (Czigler et al., 2004; Kimura et al., 2009; Stefanics et al., 2011). This difference in ERP waveforms is known as a visual mismatch response (VMR), or the visual mismatch negativity (vMMN; for recent reviews see Kimura, et al., 2011; Stefanics et al., 2014; Kremlacek et al., 2016). The vMMN is considered to be the visual counterpart of the earlier-discovered auditory MMN (Naatanen et al., 1978). VMRs are broadly theorised to reflect the tracking of patterns and regularities within sensory environments as implemented within the architecture of the visual system (e.g., Kimura et al., 2009).
Multiple phenomena can contribute to VMRs as they are measured using ERPs, each of which relates to different statistical regularities that are present in oddball sequences. The first class of phenomena relates to recent stimulation history, that is, which stimuli have recently been presented to an observer. Effects of immediate stimulus repetition, known as repetition suppression or stimulus-specific adaptation (Desimone, 1996; Movshon & Lennie, 1979), are defined as a reduction in a measure of neural activity (e.g., firing rates, local field potential amplitudes or BOLD signals) following repeated as compared to unrepeated stimuli (Grill-Spector et al., 2006). Repetition effects can partly account for VMRs as a suppression of neural responses evoked by standard stimuli and a lack of such suppression for the deviant stimuli (May & Tiitinen, 2010; Nelken & Ulanovsky, 2007). More recently developed excitatory-inhibitory circuit models also describe enhancement of responses to deviant stimuli that occur when both excitatory and lateral inhibitory responses are reduced following repeated exposure to standards (Dhruv, et al., 2011; Solomon & Kohn, 2014; Kaliukhovich & Vogels, 2016), where effects of disinhibition can occur over later time windows during the stimulus-evoked response than suppressive effects of stimulus repetition (Patterson et al., 2013). There are also systematic effects that depend on the combinations of stimuli that were recently presented to an observer (e.g., the most recently seen 2–5 stimuli, Sawamura et al., 2006; Vinken & Vogels, 2017; for similar effects on auditory evoked responses see Ulanovsky et al., 2004; Maheu et al., 2019).
The second class of phenomena relates to internally generated models of sensory environments that are learned through exposure to regularities in oddball sequences. In oddball sequences, expectations derived from these models are likely to be biased toward the often-appearing standard rather than the rare deviant. Such expectations are fulfilled when the standard is presented, and violated when a deviant appears in the expected standard's place (Friston, 2005; Stefanics et al., 2014, 2018). There is ample evidence from non-oddball designs that an observer's expectations can influence responses of stimulus-selective visual neurons (Amado et al., 2016; Feuerriegel et al., 2018a; Hall et al., 2018; Summerfield et al., 2008), and that humans can generate neural representations of expected sensory events even before the corresponding stimulus has been presented (Blom et al., 2020; Kok et al., 2017).
Within this prediction-based framework, there are two ways in which an observer's learned expectations might relate to the standard/deviant ERP differences indexed by VMRs. The first account describes VMRs as a marker of prediction error, indexing the degree of mismatch between the predictions of an observer's internal model and the actual sensory input (Friston, 2005; Garrido et al., 2009; Stefanics et al., 2014, 2018). VMRs are conceptualised as a combination of suppressed neural responses for subjectively expected standard stimuli and enhanced responses to surprising deviant stimuli. Variants of this model also incorporate the notion of precision (Feldman & Friston, 2010), whereby larger differences in the proportions of standard/deviant stimuli (and stronger weighted expectations to see standards) lead to larger prediction errors for deviant stimuli, and larger VMRs.
An alternative view is that VMRs index a surprise response that occurs when expectations have been violated, but not concurrent suppression when expectations are fulfilled. This account is equivalent to models which describe mismatch responses that occur following violations of regularities and recurring patterns in oddball sequences (e.g., Winkler, 2007; Paavilainen, 2013). Similar surprise responses also follow novel, unexpected or potentially rewarding stimuli in reward-learning paradigms, and precede the computation of reward prediction errors (reviewed in Schultz, 2016).
It has been remarkably difficult to isolate and quantify effects of immediate stimulus repetition, fulfilled expectations, and surprise using visual oddball sequences. This is because in such designs, the expected stimulus (the standard) is almost always also a repeated stimulus, whereas the surprising deviants are seldom repeated. In addition, repetition and expectation effects are likely to interact, whereby differences between expected and surprising stimuli are absent or diminished for immediately repeated stimuli (e.g., Feuerriegel et al., 2018b; Todorovic & de Lange, 2012; Wacongne et al., 2011). Recently-developed designs have attempted to overcome these limitations by adding sequences in which many different stimulus images (including the deviants and standards) are randomly interspersed within a sequence, known as equiprobable control sequences (e.g., Jacobsen & Schroger, 2001; Amado & Kovacs, 2016; reviewed in; Stefanics et al., 2014). In equiprobable sequences, each stimulus image is presented with the same probability as the deviant stimuli in the classical oddball sequences. Expectations cannot be reliably formed for any particular stimulus image in the sequence, so the stimuli are neither expected nor surprising. ERPs evoked by the same stimulus image are then compared across equiprobable and deviant contexts to isolate surprise effects while apparently controlling for immediate repetition effects.
These studies have reported separable contributions of repetition suppression and surprise to VMRs (Amado & Kovacs, 2016; Astikainen, et al., 2008; Czigler et al., 2002; Kimura et al., 2009). However, these designs do not adequately isolate effects of fulfilled expectations from repetition effects, as the expected standard stimuli are repeated frequently whereas the equiprobable control stimuli are not. There are also two additional potential confounds to consider when using this design. Massed repetition of standards is also likely to enhance responses to deviants through reductions in lateral inhibition (Dhruv et al., 2011; Kaliukhovich & Vogels, 2016), and this response enhancement would not systematically occur in equiprobable sequences. The number of stimulus identities is also different across classical oddball sequences (2 identities) and equiprobable sequences (typically 10 identities). This can produce systematic effects on visual evoked ERPs during the same time range as VMRs (Feuerriegel et al., 2018a), which have been localised to ventral temporal areas of the visual cortex (Pajani et al., 2017; Rostalski et al., 2020). It is unclear to what extent these confounds have contributed to existing reports of VMRs identified using equiprobable sequences. This has made it extremely difficult to cleanly isolate and quantify each repetition and expectation effect, which is critical for constraining theoretical models of VMRs.
To overcome these limitations, we recently developed an oddball design based on Fast Periodic Visual Stimulation (FPVS; Dzhelyova & Rossion, 2014a, 2014b; Dzhelyova, et al., 2017; Liu-Shuang, et al., 2014, 2016) that allowed us to isolate and quantify [surprising – expected] ERP differences while controlling for immediate stimulus repetition effects (Feuerriegel et al., 2018b). In FVPS oddball designs, a base stimulus is presented at a rapid, periodic rate (e.g., 6 Hz). Oddball stimuli replace the base stimulus every N stimuli at a fixed periodicity (e.g., every 7 stimuli). Viewers exposed to these stimulation sequences can perfectly predict when an oddball stimulus will appear (e.g., after 6 base rate stimuli have been presented). In sequences in which there are two possible oddball stimulus images, with one image appearing more often than the other, observers can also form expectations about the specific identity of each upcoming oddball stimulus. This leads to oddball stimulus-evoked VMRs that appear with similar topographies and latencies to VMRs found when using equiprobable sequences (Feuerriegel et al., 2018b). As six base face images are presented between each oddball face, the visual system is strongly adapted to the base face image before the presentation of each oddball stimulus in the sequences. This means that the state of adaptation in the visual system is equivalent for expected and surprising oddballs, thereby controlling for effects of immediate repetition and recent stimulation history.
In our previous work we identified [surprising – expected] response differences that were independent of stimulus repetition effects, however we could not isolate effects of surprise from those of fulfilled expectations. In the current study, we presented sets of FPVS oddball sequences and systematically varied the relative proportions of different oddball stimulus images in each sequence type, ranging from 10% to 90%. Critically, this included a 50% ‘expectation neutral’ condition, in which participants would not form strong expectations about which specific oddball stimulus was most likely to appear, as each oddball image appeared with equal probability. To isolate and quantify effects of fulfilled expectations and surprise, we compared ERPs evoked by expected (60–90% probability) and surprising (10–40% probability) oddballs with those evoked by neutral (50% probability) oddballs. We could also test whether neural responses indexed by VMRs varied in accordance with the degree of expectedness or surprise associated with seeing a given oddball stimulus (i.e., the objective oddball stimulus presentation probability).
We expected to find ERP differences for surprising compared to neutral conditions, consistent with surprise effects found in studies using equiprobable sequences (e.g., Amado et al., 2016; Kimura et al., 2009). We did not make explicit predictions regarding the magnitude of expectation suppression effects; however, we expected these to be of smaller magnitude than surprise effects, based on existing work using non-oddball designs (e.g., Amado et al., 2016). We also hypothesised that we would find linearly more negative-going ERP amplitudes for oddballs with lower presentation probabilities (i.e., more negative amplitudes for oddballs which were less probable and more surprising), consistent with the notion of precision-weighted prediction error signalling (Stefanics et al., 2014) and previous observations of graded effects on N2 and P3 ERP component amplitudes in oddball designs (e.g., Duncan-Johnson & Donchin, 1977; Polich & Margala, 1997; Wessel & Huber, 2019).
By controlling for immediate repetition effects, we could also test for adaptation effects that might occur over longer timescales, which may be important to consider in future experimental and computational modelling work. We assessed whether oddball-evoked neural response amplitudes gradually changed over the course of a stimulation sequence, as previously reported when using both classical auditory oddball (Ulanovsky et al., 2004) and FPVS oddball designs (Feuerriegel et al., 2018b).
Section snippets
Participants
We report how we determined our sample size, all data exclusions (if any), all data inclusion/exclusion criteria, whether inclusion/exclusion criteria were established prior to data analysis, all manipulations, and all measures in the study.
Forty-eight people participated in this experiment. We originally planned to recruit 50 participants so that we would retain at least 40 datasets after exclusion of excessively noisy EEG data. We did not collect the final two datasets due to budget
Task performance
Target detection rates for the fixation cross colour change detection task were near ceiling for most participants. Group mean proportion correct scores ranged between 95 and 97% across sequence types (individual participant data displayed in Supplementary Figure S1A). Group mean RTs for correctly detected targets were also very similar and ranged between 372 and 381 msec across sequence types (Supplementary Figure S1B). There was no statistically significant effect of sequence type on mean
Discussion
Using classical oddball designs it has been remarkably difficult to disentangle effects of immediate stimulus repetition, expectation and surprise in relation to visual mismatch responses (VMRs). By instead using a modified oddball design that controls for stimulus repetition effects, we could successfully isolate and quantify the contributions of expectation suppression and surprise. We report that VMRs as measured in visual oddball designs index a surprise response that occurs when an
Author contributions
Daniel Feuerriegel: conceptualisation, methodology, software, project administration, formal analysis, data curation, funding acquisition, writing - original draft.
Jane Yook: methodology, investigation software, formal analysis, resources, data curation, writing - review and editing, project administration.
Genevieve Quek: conceptualisation, methodology, supervision, writing - review and editing.
Hinze Hogendoorn: conceptualisation, methodology, supervision, writing - review and editing.
Stefan
Open practices
The study in this article earned Open Materials and Open Data badges for transparent practices. Materials and data for the study are available at https://osf.io/x8wfv/.
Acknowledgements
This project was supported by a University of Melbourne Early Career Researcher Grant awarded to D.F., a European Union Horizon 2020 Marie Sklodowska-Curie Individual Fellowship awarded to G.L.Q. (841909), and Australian Research Council (ARC) Discovery Project Grants awarded to H.H. (DP180102268) and S.B. (DP160103353). Funding sources had no role in study design, data collection, analysis or interpretation of results.
References (83)
- et al.
The contribution of surprise to the prediction based modulation of fMRI responses
Neuropsychologia
(2016) - et al.
A practical guide to the selection of independent components of the electroencephalogram for artifact correction
Journal of Neuroscience Methods
(2015) - et al.
Visual change detection: Event-related potentials are dependent on stimulus location in humans
Neuroscience Letters
(2004) - et al.
EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis
Journal of Neuroscience Methods
(2004) Selecting appropriate designs and comparison conditions in repetition paradigms
Cortex
(2016)- et al.
Perceptual expectations do not modulate image repetition effects as measured by event-related potentials
Neuroimage
(2018) - et al.
Immediate stimulus repetition abolishes stimulus expectation and surprise effects in fast periodic visual oddball designs
Biological Psychology
(2018) - et al.
The mismatch negativity: A review of underlying mechanisms
Clinical Neurophysiology
(2009) - et al.
Repetition and the brain: Neural models of stimulus-specific effects
Trends in Cognitive Sciences
(2006) - et al.
The relationship between stimulus repetitions and fulfilled expectations
Neuropsychologia
(2015)
Repetition suppression to faces in the fusiform face area: A personal and dynamic journey
Cortex
The effect of repetition lag on electrophysiological and haemodynamic correlates of visual object priming
Neuroimage
Impaired mismatch negativity (MMN) generation in schizophrenia as a function of stimulus deviance, probability, and interstimulus/interdeviant interval
Clinical Neurophysiology
Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects
Clinical Neurophysiology
Visual mismatch negativity (vMMN): A review and meta-analysis of studies in psychiatric and neurological disorders
Cortex
An objective index of individual face discrimination in the right occipito-temporal cortex by means of fast periodic oddball stimulation
Neuropsychologia
An objective electrophysiological marker of face individualisation impairment in acquired prosopagnosia with fast periodic visual stimulation
Neuropsychologia
Nonparametric statistical testing of EEG- and MEG-data
Journal of Neuroscience Methods
Early selective-attention effect on evoked potential reinterpreted
Acta Psychologica
The mismatch-negativity (MMN) component of the auditory event-related potential to violations of abstract regularities: A review
International Journal of Psychophysiology
Effect of deviant probability and interstimulus/interdeviant interval on the auditory N1 and mismatch negativity in the cat auditory cortex
Cognitive Brain Research
P300 and probability: Comparisons of oddball and single-stimulus paradigms
International Journal of Psychophysiology
Category-selective human brain processes elicitied in fast periodic visual stimulation streams are immune to temporal predictability
Neuropsychologia
Measures of repetition suppression in the Fusiform Face Area are inflated by co-occurring effects of statistically learned visual associations
Cortex
Selectivity of neuronal adaptation does not match response selectivity: A single-cell study of the fMRI adaptation paradigm
Neuron
Short- and long-lasting consequences of novelty, deviance and surprise on brain and cognition
Neuroscience and Biobehavioural Reviews
Global effects of feature-based attention depend on surprise
Neuroimage
Moving sensory adaptation beyond suppressive effects in single neurons
Current Biology
Feature-specific prediction errors for visual mismatch
Neuroimage
Does surprise enhancement or repetition suppression explain visual mismatch negativity?
European Journal of Neuroscience
Visual mismatch negativity for changes in orientation - a sensory memory-dependent response
European Journal of Neuroscience
Controlling the false discovery rate: A practical and powerful approach to multiple testing
Journal of the Royal Statistical Society Series B: Statistical Methodology
Single-trial event-related potential correlates of belief updating
eNeuro
Predictions drive neural representations of visual events ahead of incoming sensory information
Proceedings of the National Academy of Sciences of the United States of America
The psychophysics toolbox
Spatial Vision
How does noise affect amplitude and latency measurement of event-related potentials (ERPs)? A methodological critique and simulation study
Psychophysiology
Memory-based detection of task-irrelevant visual changes
Psychophysiology
Effects of adaptation on the stimulus selectivity of macaque inferior temporal spiking activity and local field potentials
Cerebral Cortex
Neural mechanisms for visual memory and their role in attention
Proceedings of the National Academy of Sciences of the United States of America
Multiple adaptable mechanisms early in the primate visual pathway
Journal of Neuroscience
On quantifying surprise: The variation of event-related potentials with subjective probability
Psychophysiology
Cited by (14)
Sixty years of predictive perception
2024, CortexEffects of iron supplementation on neural indices of habituation in Bangladeshi children
2023, American Journal of Clinical NutritionViolated expectations for spatial and feature attributes of visual trajectories modulate event-related potential amplitudes across the visual processing hierarchy
2022, Biological PsychologyCitation Excerpt :Thirdly, while the “oddball-ness” of deviant stimuli in the oddball paradigms is defined through all trials across an experiment (e.g., the deviant stimuli appear in 10 % of the trials), it was established within each trial in the contextual trajectory paradigm. As a result, our study identified ERP responses that reflected local violations within four-step sequences, whereas the vMMN should be more reflective of global violations in whole blocks of trials (Feuerriegel et al., 2021; Stefanics et al., 2014). Overall, our paradigm offers novelty over the vMMN literature as we demonstrate within one experiment that the early visual evoked ERPs – the N170 and N300 (and potentially the P1pc too) – are all indexing different aspects of a predictive coding hierarchy, which are key components of the iterative nature of a predictive coding model.
Electrophysiological correlates of confidence differ across correct and erroneous perceptual decisions
2022, NeuroImageCitation Excerpt :This was done separately for analyses of trials with correct responses and trials with errors. The resulting Beta coefficients (slopes) were tested at the group-level using one-sample frequentist and Bayesian t-tests (as done by Feuerriegel et al., 2021b). For analyses of the CPP the correct/error comparison included both trials whereby the S2 target did and did not appear, as the stimulation conditions were not systematically different until the time of the response.