Elsevier

Cortex

Volume 134, January 2021, Pages 16-29
Cortex

Research Report
Visual mismatch responses index surprise signalling but not expectation suppression

https://doi.org/10.1016/j.cortex.2020.10.006Get rights and content

Abstract

The ability to distinguish between commonplace and unusual sensory events is critical for efficient learning and adaptive behaviour. This has been investigated using oddball designs in which sequences of often-appearing (i.e., expected) stimuli are interspersed with rare (i.e., surprising) deviants. Resulting differences in electrophysiological responses following surprising compared to expected stimuli are known as visual mismatch responses (VMRs). VMRs are thought to index co-occurring contributions of stimulus repetition effects, expectation suppression (that occurs when one's expectations are fulfilled), and expectation violation (i.e., surprise) responses; however, these different effects have been conflated in existing oddball designs. To better isolate and quantify effects of expectation suppression and surprise, we adapted an oddball design based on Fast Periodic Visual Stimulation (FPVS) that controls for stimulus repetition effects. We recorded electroencephalography (EEG) while participants (N = 48) viewed stimulation sequences in which a single face identity was periodically presented at 6 Hz. Critically, one of two different face identities (termed oddballs) appeared as every 7th image throughout the sequence. The presentation probabilities of each oddball image within a sequence varied between 10 and 90%, such that participants could form expectations about which oddball face identity was more likely to appear within each sequence. We also included ‘expectation neutral’ 50% probability sequences, whereby consistently biased expectations would not be formed for either oddball face identity. We found that VMRs indexed surprise responses, and effects of expectation suppression were absent. That is, ERPs were more negative-going at occipitoparietal electrodes for surprising compared to neutral oddballs, but did not differ between expected and neutral oddballs. Surprising oddball-evoked ERPs were also highly similar across the 10–40% appearance probability conditions. Our findings indicate that VMRs which are not accounted for by repetition effects are best described as an all-or-none surprise response, rather than a minimisation of prediction error responses associated with expectation suppression.

Introduction

We are highly adept at learning recurring patterns and statistical regularities that occur in our sensory environments, and we can exploit this knowledge to discriminate between commonplace and unusual sensory events. The computations that underpin this capability likely rely on statistics learned over time-scales ranging from hundreds of milliseconds to several minutes (Maheu et al., 2019; Ulanovsky et al., 2004). Characterising the computations that occur within different brain regions can help us understand how we generate and evaluate internal models of our environment, and how these models are used to facilitate detection of potential rewards (Schultz, 2016) and enact adaptive adjustments to decision-making strategies (Wessel, 2017; Wessel & Huber, 2019).

Detection of novel or unusual events has been investigated using visual oddball designs in which a stimulus is presented in high and low probability contexts. For example, in the high probability context, a critical stimulus A is presented frequently as the ‘standard’ stimulus, interspersed with a rare ‘deviant’ stimulus B (e.g., AABAAAAAAABAAAB). In the low probability context, the standard stimulus A is instead presented as a rare deviant (e.g., BBBBBABBBBBABBB). Comparisons of visual stimulus-evoked event-related potentials (ERPs) recorded using electroencephalography (EEG) reveal more negative-going waveforms evoked by deviants at posterior scalp electrodes from 150 to 300 msec following stimulus onset (Czigler et al., 2004; Kimura et al., 2009; Stefanics et al., 2011). This difference in ERP waveforms is known as a visual mismatch response (VMR), or the visual mismatch negativity (vMMN; for recent reviews see Kimura, et al., 2011; Stefanics et al., 2014; Kremlacek et al., 2016). The vMMN is considered to be the visual counterpart of the earlier-discovered auditory MMN (Naatanen et al., 1978). VMRs are broadly theorised to reflect the tracking of patterns and regularities within sensory environments as implemented within the architecture of the visual system (e.g., Kimura et al., 2009).

Multiple phenomena can contribute to VMRs as they are measured using ERPs, each of which relates to different statistical regularities that are present in oddball sequences. The first class of phenomena relates to recent stimulation history, that is, which stimuli have recently been presented to an observer. Effects of immediate stimulus repetition, known as repetition suppression or stimulus-specific adaptation (Desimone, 1996; Movshon & Lennie, 1979), are defined as a reduction in a measure of neural activity (e.g., firing rates, local field potential amplitudes or BOLD signals) following repeated as compared to unrepeated stimuli (Grill-Spector et al., 2006). Repetition effects can partly account for VMRs as a suppression of neural responses evoked by standard stimuli and a lack of such suppression for the deviant stimuli (May & Tiitinen, 2010; Nelken & Ulanovsky, 2007). More recently developed excitatory-inhibitory circuit models also describe enhancement of responses to deviant stimuli that occur when both excitatory and lateral inhibitory responses are reduced following repeated exposure to standards (Dhruv, et al., 2011; Solomon & Kohn, 2014; Kaliukhovich & Vogels, 2016), where effects of disinhibition can occur over later time windows during the stimulus-evoked response than suppressive effects of stimulus repetition (Patterson et al., 2013). There are also systematic effects that depend on the combinations of stimuli that were recently presented to an observer (e.g., the most recently seen 2–5 stimuli, Sawamura et al., 2006; Vinken & Vogels, 2017; for similar effects on auditory evoked responses see Ulanovsky et al., 2004; Maheu et al., 2019).

The second class of phenomena relates to internally generated models of sensory environments that are learned through exposure to regularities in oddball sequences. In oddball sequences, expectations derived from these models are likely to be biased toward the often-appearing standard rather than the rare deviant. Such expectations are fulfilled when the standard is presented, and violated when a deviant appears in the expected standard's place (Friston, 2005; Stefanics et al., 2014, 2018). There is ample evidence from non-oddball designs that an observer's expectations can influence responses of stimulus-selective visual neurons (Amado et al., 2016; Feuerriegel et al., 2018a; Hall et al., 2018; Summerfield et al., 2008), and that humans can generate neural representations of expected sensory events even before the corresponding stimulus has been presented (Blom et al., 2020; Kok et al., 2017).

Within this prediction-based framework, there are two ways in which an observer's learned expectations might relate to the standard/deviant ERP differences indexed by VMRs. The first account describes VMRs as a marker of prediction error, indexing the degree of mismatch between the predictions of an observer's internal model and the actual sensory input (Friston, 2005; Garrido et al., 2009; Stefanics et al., 2014, 2018). VMRs are conceptualised as a combination of suppressed neural responses for subjectively expected standard stimuli and enhanced responses to surprising deviant stimuli. Variants of this model also incorporate the notion of precision (Feldman & Friston, 2010), whereby larger differences in the proportions of standard/deviant stimuli (and stronger weighted expectations to see standards) lead to larger prediction errors for deviant stimuli, and larger VMRs.

An alternative view is that VMRs index a surprise response that occurs when expectations have been violated, but not concurrent suppression when expectations are fulfilled. This account is equivalent to models which describe mismatch responses that occur following violations of regularities and recurring patterns in oddball sequences (e.g., Winkler, 2007; Paavilainen, 2013). Similar surprise responses also follow novel, unexpected or potentially rewarding stimuli in reward-learning paradigms, and precede the computation of reward prediction errors (reviewed in Schultz, 2016).

It has been remarkably difficult to isolate and quantify effects of immediate stimulus repetition, fulfilled expectations, and surprise using visual oddball sequences. This is because in such designs, the expected stimulus (the standard) is almost always also a repeated stimulus, whereas the surprising deviants are seldom repeated. In addition, repetition and expectation effects are likely to interact, whereby differences between expected and surprising stimuli are absent or diminished for immediately repeated stimuli (e.g., Feuerriegel et al., 2018b; Todorovic & de Lange, 2012; Wacongne et al., 2011). Recently-developed designs have attempted to overcome these limitations by adding sequences in which many different stimulus images (including the deviants and standards) are randomly interspersed within a sequence, known as equiprobable control sequences (e.g., Jacobsen & Schroger, 2001; Amado & Kovacs, 2016; reviewed in; Stefanics et al., 2014). In equiprobable sequences, each stimulus image is presented with the same probability as the deviant stimuli in the classical oddball sequences. Expectations cannot be reliably formed for any particular stimulus image in the sequence, so the stimuli are neither expected nor surprising. ERPs evoked by the same stimulus image are then compared across equiprobable and deviant contexts to isolate surprise effects while apparently controlling for immediate repetition effects.

These studies have reported separable contributions of repetition suppression and surprise to VMRs (Amado & Kovacs, 2016; Astikainen, et al., 2008; Czigler et al., 2002; Kimura et al., 2009). However, these designs do not adequately isolate effects of fulfilled expectations from repetition effects, as the expected standard stimuli are repeated frequently whereas the equiprobable control stimuli are not. There are also two additional potential confounds to consider when using this design. Massed repetition of standards is also likely to enhance responses to deviants through reductions in lateral inhibition (Dhruv et al., 2011; Kaliukhovich & Vogels, 2016), and this response enhancement would not systematically occur in equiprobable sequences. The number of stimulus identities is also different across classical oddball sequences (2 identities) and equiprobable sequences (typically 10 identities). This can produce systematic effects on visual evoked ERPs during the same time range as VMRs (Feuerriegel et al., 2018a), which have been localised to ventral temporal areas of the visual cortex (Pajani et al., 2017; Rostalski et al., 2020). It is unclear to what extent these confounds have contributed to existing reports of VMRs identified using equiprobable sequences. This has made it extremely difficult to cleanly isolate and quantify each repetition and expectation effect, which is critical for constraining theoretical models of VMRs.

To overcome these limitations, we recently developed an oddball design based on Fast Periodic Visual Stimulation (FPVS; Dzhelyova & Rossion, 2014a, 2014b; Dzhelyova, et al., 2017; Liu-Shuang, et al., 2014, 2016) that allowed us to isolate and quantify [surprising – expected] ERP differences while controlling for immediate stimulus repetition effects (Feuerriegel et al., 2018b). In FVPS oddball designs, a base stimulus is presented at a rapid, periodic rate (e.g., 6 Hz). Oddball stimuli replace the base stimulus every N stimuli at a fixed periodicity (e.g., every 7 stimuli). Viewers exposed to these stimulation sequences can perfectly predict when an oddball stimulus will appear (e.g., after 6 base rate stimuli have been presented). In sequences in which there are two possible oddball stimulus images, with one image appearing more often than the other, observers can also form expectations about the specific identity of each upcoming oddball stimulus. This leads to oddball stimulus-evoked VMRs that appear with similar topographies and latencies to VMRs found when using equiprobable sequences (Feuerriegel et al., 2018b). As six base face images are presented between each oddball face, the visual system is strongly adapted to the base face image before the presentation of each oddball stimulus in the sequences. This means that the state of adaptation in the visual system is equivalent for expected and surprising oddballs, thereby controlling for effects of immediate repetition and recent stimulation history.

In our previous work we identified [surprising – expected] response differences that were independent of stimulus repetition effects, however we could not isolate effects of surprise from those of fulfilled expectations. In the current study, we presented sets of FPVS oddball sequences and systematically varied the relative proportions of different oddball stimulus images in each sequence type, ranging from 10% to 90%. Critically, this included a 50% ‘expectation neutral’ condition, in which participants would not form strong expectations about which specific oddball stimulus was most likely to appear, as each oddball image appeared with equal probability. To isolate and quantify effects of fulfilled expectations and surprise, we compared ERPs evoked by expected (60–90% probability) and surprising (10–40% probability) oddballs with those evoked by neutral (50% probability) oddballs. We could also test whether neural responses indexed by VMRs varied in accordance with the degree of expectedness or surprise associated with seeing a given oddball stimulus (i.e., the objective oddball stimulus presentation probability).

We expected to find ERP differences for surprising compared to neutral conditions, consistent with surprise effects found in studies using equiprobable sequences (e.g., Amado et al., 2016; Kimura et al., 2009). We did not make explicit predictions regarding the magnitude of expectation suppression effects; however, we expected these to be of smaller magnitude than surprise effects, based on existing work using non-oddball designs (e.g., Amado et al., 2016). We also hypothesised that we would find linearly more negative-going ERP amplitudes for oddballs with lower presentation probabilities (i.e., more negative amplitudes for oddballs which were less probable and more surprising), consistent with the notion of precision-weighted prediction error signalling (Stefanics et al., 2014) and previous observations of graded effects on N2 and P3 ERP component amplitudes in oddball designs (e.g., Duncan-Johnson & Donchin, 1977; Polich & Margala, 1997; Wessel & Huber, 2019).

By controlling for immediate repetition effects, we could also test for adaptation effects that might occur over longer timescales, which may be important to consider in future experimental and computational modelling work. We assessed whether oddball-evoked neural response amplitudes gradually changed over the course of a stimulation sequence, as previously reported when using both classical auditory oddball (Ulanovsky et al., 2004) and FPVS oddball designs (Feuerriegel et al., 2018b).

Section snippets

Participants

We report how we determined our sample size, all data exclusions (if any), all data inclusion/exclusion criteria, whether inclusion/exclusion criteria were established prior to data analysis, all manipulations, and all measures in the study.

Forty-eight people participated in this experiment. We originally planned to recruit 50 participants so that we would retain at least 40 datasets after exclusion of excessively noisy EEG data. We did not collect the final two datasets due to budget

Task performance

Target detection rates for the fixation cross colour change detection task were near ceiling for most participants. Group mean proportion correct scores ranged between 95 and 97% across sequence types (individual participant data displayed in Supplementary Figure S1A). Group mean RTs for correctly detected targets were also very similar and ranged between 372 and 381 msec across sequence types (Supplementary Figure S1B). There was no statistically significant effect of sequence type on mean

Discussion

Using classical oddball designs it has been remarkably difficult to disentangle effects of immediate stimulus repetition, expectation and surprise in relation to visual mismatch responses (VMRs). By instead using a modified oddball design that controls for stimulus repetition effects, we could successfully isolate and quantify the contributions of expectation suppression and surprise. We report that VMRs as measured in visual oddball designs index a surprise response that occurs when an

Author contributions

Daniel Feuerriegel: conceptualisation, methodology, software, project administration, formal analysis, data curation, funding acquisition, writing - original draft.

Jane Yook: methodology, investigation software, formal analysis, resources, data curation, writing - review and editing, project administration.

Genevieve Quek: conceptualisation, methodology, supervision, writing - review and editing.

Hinze Hogendoorn: conceptualisation, methodology, supervision, writing - review and editing.

Stefan

Open practices

The study in this article earned Open Materials and Open Data badges for transparent practices. Materials and data for the study are available at https://osf.io/x8wfv/.

Acknowledgements

This project was supported by a University of Melbourne Early Career Researcher Grant awarded to D.F., a European Union Horizon 2020 Marie Sklodowska-Curie Individual Fellowship awarded to G.L.Q. (841909), and Australian Research Council (ARC) Discovery Project Grants awarded to H.H. (DP180102268) and S.B. (DP160103353). Funding sources had no role in study design, data collection, analysis or interpretation of results.

References (83)

  • R.N. Henson

    Repetition suppression to faces in the fusiform face area: A personal and dynamic journey

    Cortex

    (2016)
  • R.N. Henson et al.

    The effect of repetition lag on electrophysiological and haemodynamic correlates of visual object priming

    Neuroimage

    (2004)
  • D.C. Javitt et al.

    Impaired mismatch negativity (MMN) generation in schizophrenia as a function of stimulus deviance, probability, and interstimulus/interdeviant interval

    Clinical Neurophysiology

    (1998)
  • T.P. Jung et al.

    Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects

    Clinical Neurophysiology

    (2000)
  • J. Kremlacek et al.

    Visual mismatch negativity (vMMN): A review and meta-analysis of studies in psychiatric and neurological disorders

    Cortex

    (2016)
  • J. Liu-Shuang et al.

    An objective index of individual face discrimination in the right occipito-temporal cortex by means of fast periodic oddball stimulation

    Neuropsychologia

    (2014)
  • J. Liu-Shuang et al.

    An objective electrophysiological marker of face individualisation impairment in acquired prosopagnosia with fast periodic visual stimulation

    Neuropsychologia

    (2016)
  • E. Maris et al.

    Nonparametric statistical testing of EEG- and MEG-data

    Journal of Neuroscience Methods

    (2007)
  • R. Naatanen et al.

    Early selective-attention effect on evoked potential reinterpreted

    Acta Psychologica

    (1978)
  • P. Paavilainen

    The mismatch-negativity (MMN) component of the auditory event-related potential to violations of abstract regularities: A review

    International Journal of Psychophysiology

    (2013)
  • Z. Pincze et al.

    Effect of deviant probability and interstimulus/interdeviant interval on the auditory N1 and mismatch negativity in the cat auditory cortex

    Cognitive Brain Research

    (2002)
  • J. Polich et al.

    P300 and probability: Comparisons of oddball and single-stimulus paradigms

    International Journal of Psychophysiology

    (1997)
  • G.L. Quek et al.

    Category-selective human brain processes elicitied in fast periodic visual stimulation streams are immune to temporal predictability

    Neuropsychologia

    (2017)
  • S.-M. Rostalski et al.

    Measures of repetition suppression in the Fusiform Face Area are inflated by co-occurring effects of statistically learned visual associations

    Cortex

    (October 2020)
  • H. Sawamura et al.

    Selectivity of neuronal adaptation does not match response selectivity: A single-cell study of the fMRI adaptation paradigm

    Neuron

    (2006)
  • J. Schomaker et al.

    Short- and long-lasting consequences of novelty, deviance and surprise on brain and cognition

    Neuroscience and Biobehavioural Reviews

    (2015)
  • C. Smout et al.

    Global effects of feature-based attention depend on surprise

    Neuroimage

    (2020)
  • S.G. Solomon et al.

    Moving sensory adaptation beyond suppressive effects in single neurons

    Current Biology

    (2014)
  • G. Stefanics et al.

    Feature-specific prediction errors for visual mismatch

    Neuroimage

    (2019)
  • C. Amado et al.

    Does surprise enhancement or repetition suppression explain visual mismatch negativity?

    European Journal of Neuroscience

    (2016)
  • P. Astikainen et al.

    Visual mismatch negativity for changes in orientation - a sensory memory-dependent response

    European Journal of Neuroscience

    (2008)
  • Y. Benjamini et al.

    Controlling the false discovery rate: A practical and powerful approach to multiple testing

    Journal of the Royal Statistical Society Series B: Statistical Methodology

    (1995)
  • D. Bennett et al.

    Single-trial event-related potential correlates of belief updating

    eNeuro

    (2015)
  • T. Blom et al.

    Predictions drive neural representations of visual events ahead of incoming sensory information

    Proceedings of the National Academy of Sciences of the United States of America

    (2020)
  • D.H. Brainard

    The psychophysics toolbox

    Spatial Vision

    (1997)
  • P.E. Clayson et al.

    How does noise affect amplitude and latency measurement of event-related potentials (ERPs)? A methodological critique and simulation study

    Psychophysiology

    (2012)
  • I. Czigler et al.

    Memory-based detection of task-irrelevant visual changes

    Psychophysiology

    (2002)
  • W. De Baene et al.

    Effects of adaptation on the stimulus selectivity of macaque inferior temporal spiking activity and local field potentials

    Cerebral Cortex

    (2010)
  • R. Desimone

    Neural mechanisms for visual memory and their role in attention

    Proceedings of the National Academy of Sciences of the United States of America

    (1996)
  • N.T. Dhruv et al.

    Multiple adaptable mechanisms early in the primate visual pathway

    Journal of Neuroscience

    (2011)
  • C.C. Duncan-Johnson et al.

    On quantifying surprise: The variation of event-related potentials with subjective probability

    Psychophysiology

    (1977)
  • Cited by (14)

    • Violated expectations for spatial and feature attributes of visual trajectories modulate event-related potential amplitudes across the visual processing hierarchy

      2022, Biological Psychology
      Citation Excerpt :

      Thirdly, while the “oddball-ness” of deviant stimuli in the oddball paradigms is defined through all trials across an experiment (e.g., the deviant stimuli appear in 10 % of the trials), it was established within each trial in the contextual trajectory paradigm. As a result, our study identified ERP responses that reflected local violations within four-step sequences, whereas the vMMN should be more reflective of global violations in whole blocks of trials (Feuerriegel et al., 2021; Stefanics et al., 2014). Overall, our paradigm offers novelty over the vMMN literature as we demonstrate within one experiment that the early visual evoked ERPs – the N170 and N300 (and potentially the P1pc too) – are all indexing different aspects of a predictive coding hierarchy, which are key components of the iterative nature of a predictive coding model.

    • Electrophysiological correlates of confidence differ across correct and erroneous perceptual decisions

      2022, NeuroImage
      Citation Excerpt :

      This was done separately for analyses of trials with correct responses and trials with errors. The resulting Beta coefficients (slopes) were tested at the group-level using one-sample frequentist and Bayesian t-tests (as done by Feuerriegel et al., 2021b). For analyses of the CPP the correct/error comparison included both trials whereby the S2 target did and did not appear, as the stimulation conditions were not systematically different until the time of the response.

    View all citing articles on Scopus
    View full text