A Bayesian brain model of adaptive behavior: an application to the Wisconsin Card Sorting Task

The model

The core idea behind our computational framework is to encode the concept of belief into a generative probabilistic model of the environment. Belief updating then corresponds to recursive Bayesian updating of the internal model based on current and past interactions between the agent and its environment. Optimal or sub-optimal actions are selected according to a well specified or a misspecified internal model and, in turn, cause perceptible changes in the environment.

We assume that the cognitive agent aims to infer the true hidden state of the environment by processing and integrating sensory information from the environment. Within the context of the WCST, the hidden environmental states might change as a function of both the structure of the task and the (often sub-optimal) behavioral dynamics, so the agent constantly needs to rely on environmental feedback and own actions to infer the current state. We assume that the agent maintains an internal probability distribution over the states at each individual trial of the WCST. The agent then updates this distribution upon making new observations. In particular, the hidden environmental states to be inferred are the three features, s_t ∈ {1, 2, 3}, which refer the three possible sorting rules in the task environment such that 1: color, 2: shape and 3: number of objects. The posterior probability of the states depends on an observation vector x_t = (a_t, f_t), which consists of the pair of agent’s response a_t ∈ {1, 2, 3, 4}, codifying the action of choosing deck 1, 2, 3 or 4, and received feedback f_t ∈ {0, 1}, referring to the fact that a given response results in a failure (0) or in a success (1), in a given trial t = 0, …, T. The discrete response a_t represents the stimulus card indicator being matched with a target card at trial t. We denote a sequence of observations as x_0:t = (x₀, x₁, …, x_t) = ((a₀, f₀), (a₁, f₁), (a₂, f₂), …, (a_t, f_t)) and set x₀ = ∅ in order to indicate that there are no observations at the onset of the task. Thus, trial-by-trial belief updating is recursively computed according to Bayes’ rule: (1)${\displaystyle p\left(s_t|x_{0:t}\right)=\frac{p\left(x_t|s_t,x_{0:t-1}\right)p\left(s_t|x_{0:t-1}\right)}{p\left(x_t|x_{0:t-1}\right)}.}$

Accordingly, the agent’s posterior belief about the task-relevant features s_t after observing a sequence of response-feedback pairs x_0:t is proportional to the product of the likelihood of observing a particular response-feedback pair and the agent’s prior belief about the task-relevant feature in the current trial. The likelihood of an observation is computed as follows: (2)${\displaystyle p\left(x_t|s_t,x_{0:t-1}\right)=\frac{f_{t}p\left(a_t|s_t=i\right)+\left(1-f_t\right)\left(1-p\left(a_t|s_t=i\right)\right)}{f_t\sum _{j}p\left(a_t|s_t=j\right)+\left(1-f_t\right)\sum _j\left(1-p\left(a_t|s_t=j\right)\right)}}$

where j = 1, 2, 3 and p(a_t|s_t = i) indicates the probability of a matching between the target and the stimulus card assumed that the current feature is i. Here, we assume the likelihood of a current observation to be independent from previous observations without loss of generality, that is: ${\displaystyle p\left(x_t|s_t,x_{0:t-1}\right)=p\left(x_t|s_t\right).}$

The prior belief for a given trial t is computed based on the posterior belief generated in the previous trial, p(s_t−1|x_0:t−1), and the agent’s belief about the probability of transitions between the hidden states, p(s_t|s_t−1). The prior belief can also be considered as a predictive probability over the hidden states. The predictive distribution for an upcoming trial t is computed according to the Chapman–Kolmogorov equation: (3)${\displaystyle p\left(s_{t+1}=k|x_{0:t}\right)=\sum _{i=1}^{3}p\left(s_{t+1}=k|s_t=i,\Gamma \left(t\right)\right)p\left(s_t=i|x_{0:t}\right)}$

where Γ(t) represents a stability matrix describing transitions between the states (to be explained shortly). Thus, the agent combines information from the updated belief (posterior distribution) and the belief about the transition properties of the environmental states to predict the most probable future state. The predictive distribution represents the internal model of the cognitive agent according to which actions are generated.

The stability matrix Γ(t) encodes the agent’s belief about the probability of states being stable or likely to change in the next trial. In other words, the stability matrix reflects the cognitive agent’s internal representation of the dynamic probabilistic model of the task environment. It is computed on each trial based on the response-feedback pair, x_t, and a matching signal, m_t, which are observed.

The matching signal m_t is a vector informing the cognitive agent which features are currently relevant (meaningful), such that $m_t^{\left(i\right)}=1$ when a positive feedback is associated with a response implying feature s_t = i, and $m_t^{\left(i\right)}=0$ otherwise. Note, that the matching signal is not a free parameter of the model, but is completely determined by the task contingencies. The matching signal vector allows the agent to compute the state activation level ${\omega }_t^{\left(i\right)}\in \left[0,1\right]$ for the hidden state s_t = i, which provides an internal measure of the (accumulated) evidence for each hidden state at trial t. Thus, the activation levels of the hidden states are represented by a vector ω_t. The stability matrix is a square and asymmetric matrix related to hidden state activation levels such that: (4)${\displaystyle \Gamma \left(t\right)=\left[\begin{array}{c}\hfill {\omega }_t^{\left(1\right)}\frac{1}{2}\left(1-{\omega }_t^{\left(1\right)}\right)\frac{1}{2}\left(1-{\omega }_t^{\left(1\right)}\right)\hfill \\ \hfill \frac{1}{2}\left(1-{\omega }_t^{\left(2\right)}\right){\omega }_t^{\left(2\right)}\frac{1}{2}\left(1-{\omega }_t^{\left(2\right)}\right)\hfill \\ \hfill \frac{1}{2}\left(1-{\omega }_t^{\left(3\right)}\right)\frac{1}{2}\left(1-{\omega }_t^{\left(3\right)}\right){\omega }_t^{\left(3\right)}\hfill \end{array}\right]}$

where the entries Γ_ii(t) in the main diagonal represent the elements of the activation vector ω_t, and the non-diagonal elements are computed so as to ensure that rows sum to 1. The state activation vector is computed in each trial as follows: (5)${\displaystyle \left[\begin{array}{c}\hfill {\omega }_t^{\left(1\right)}\hfill \\ \hfill {\omega }_t^{\left(2\right)}\hfill \\ \hfill {\omega }_t^{\left(3\right)}\hfill \end{array}\right]=f_t{\omega }_{t-1}^{\delta }\left[\begin{array}{c}\hfill m_t^{\left(1\right)}\hfill \\ \hfill m_t^{\left(2\right)}\hfill \\ \hfill m_t^{\left(3\right)}\hfill \end{array}\right]+\lambda \left[\left(1-f_t\right){\omega }_{t-1}^{\delta }\left[\begin{array}{c}\hfill 1-m_t^{\left(1\right)}\hfill \\ \hfill 1-m_t^{\left(2\right)}\hfill \\ \hfill 1-m_t^{\left(3\right)}\hfill \end{array}\right]\right]\left[\begin{array}{c}\hfill {\omega }_{t-1}^{\left(1\right)}\hfill \\ \hfill {\omega }_{t-1}^{\left(2\right)}\hfill \\ \hfill {\omega }_{t-1}^{\left(3\right)}\hfill \end{array}\right].}$

This equation reflects the idea that state activations are simultaneously affected by the observed feedback, f_t, and the matching signal vector, m_t. However, the matching signal vector conveys different information based on the current feedback. Matching a target card with a stimulus card makes a feature (or a subset of features) informative for a specific state. The vector m_t contributes to increase the activation level of a state if the feature is informative for that state when a positive feedback is received, as well as to decrease the activation level when a negative feedback is received.

The parameter λ ∈ [0, 1] modulates the efficiency to disengage attention to a given state-activation configuration when a negative feedback is processed. We therefore term this parameter flexibility. We also assume that information from the matching signal vector can degrade by slowing down the rate of evidence accumulation for the hidden states. This means that the matching signal vector can be re-scaled based on the current state activation level. The parameter δ ∈ [0, 1] is introduced to achieve this re-scaling. When δ = 0, there is no re-scaling and updating of the state activation levels relies on the entire information conveyed by m_t. On the other extreme, when δ = 1, several trials have to be accomplished before converging to a given configuration of the state activation levels. Equivalently, higher values of δ affect the entropy of the distribution over hidden states by decreasing the probability of sampling of the correct feature. We therefore refer to δ as information loss.

The free parameters λ and δ are central to our computational model, since they regulate the rate at which the internal model converges to the true task environmental model. can be expressed in compact notation as follows: (6)${\displaystyle {\omega }_t=f_t{\omega }_{t-1}^{\delta }{m}_t+\lambda \left[\left(1-f_t\right){\omega }_{t-1}^{\delta }\left(1-m_t\right)\right]{\omega }_{t-1}.}$

Note that the information loss parameter δ affects the amount of information that a cognitive agent acquires from environmental contingencies, irrespective of the type of feedback received. Global information loss thus affects the rate at which the divergence between the agent’s internal model and the true model is minimized. Figure 2 illustrates these ideas.

The probabilistic representation of adaptive behaviour provided by our Bayesian agent model allows us to quantify latent cognitive dynamics by means of meaningful information-theoretic measures. Information theory has proven to be an effective and natural mathematical language to account for functional integration of structured cognitive processes and to relate them to brain activity (Koechlin & Summerfield, 2007; Friston et al., 2017; Collell & Fauquet, 2015; Strange et al., 2005; Friston, 2003). In particular, we are interested in three key measures, namely, Bayesian surprise, ${\mathcal{B}}_t$, Shannon surprise, ${\mathcal{I}}_t$, and entropy, ${\mathcal{H}}_t$. The subscript t indicates that we can compute each quantity on a trial-by-trial basis. Each quantity is amenable to a specific interpretation in terms of separate neurocognitive processes. Bayesian surprise ${\mathcal{B}}_t$ quantifies the magnitude of the update from prior belief to posterior belief. Shannon surprise ${\mathcal{I}}_t$ quantifies the improbability of an observation given an agent’s prior expectation. Finally, entropy ${\mathcal{H}}_t$ measures the degree of epistemic uncertainty regarding the true environmental states. Such measures are thought to account for the ability of the agent to manage uncertainty as emerging as a function of competing behavioral affordances (Hirsh, Mar & Peterson, 2012). We expect an adaptive system to attenuate uncertainty over environmental states (current features) by reducing the entropy of its internal probabilistic model.

Bayesian surprise can be computed as the Kullback–Leibler (KL) divergence between prior and posterior beliefs about the environmental states. Thus, Bayesian surprise accounts for the divergence between the predictive model for the current trial and the updated predictive model for the upcoming trial. It is computed as follows: (7)${\displaystyle {\mathcal{B}}_t=\mathbb{KL}\left[p\left(s_{t+1}|x_{0:t}\right)\left|\right|p\left(s_t|x_{0:t-1}\right)\right]=\sum _{i=1}^3\left[p\left(s_{t+1}=i|x_{0:t}\right)log\left(\frac{p\left(s_{t+1}=i|x_{0:t}\right)}{p\left(s_t=i|x_{0:t-1}\right)}\right)\right]}$

The Shannon surprise of a current observation given a previous one is computed as the conditional information content of the observation: (8)${\displaystyle {\mathcal{I}}_t=-logp\left(x_t|x_{0:t-1}\right)=-log\sum _{i=1}^3\left[p\left(x_t|s_t=i\right)p\left(s_t=i|x_{0:t-1}\right)\right]}$

Finally, the entropy is computed over the predictive distribution in order to account for the uncertainty in the internal model of the agent in trial t as follows: (9)${\displaystyle {\mathcal{H}}_t=\mathbb{E}\left[-logp\left(s_t|x_{0:t-1}\right)\right]=-\sum _{i=1}^{3}p\left(s_t=i|x_{0:t-1}\right)logp\left(s_t=i|x_{0:t-1}\right)}$

Once the flexibility (λ) and information loss (δ) parameters are estimated from data, the information-theoretic quantities can be easily computed and visualized for each trial of the WCST (see Fig. 2). This allows to rephrase standard neurocognitive constructs in terms of measurable information-theoretic quantities. Moreover, the dynamics of these quantities, as well as their interactions, can be used for formulating and testing hypotheses about the neurcognitive underpinnings of adaptive behavior in a principled way, as discussed later in the paper. A summary of all quantities relevant for our computational model is provided in Table 1.

Simulations

In this section we evaluate the expressiveness of the model by assessing its ability to reproduce meaningful behavioral patterns as a function of its two free parameters. We study how the generative model behaves when performing the WCST in a 2-factorial simulated Monte Carlo design where flexibility (λ) and information loss (δ) are systematically varied.

In this simulation, the Heaton version of the task (Heaton, 1981) is administered to the Bayesian cognitive agent. In this particular version, the sorting rule (true environmental state) changes after a fixed number of consecutive correct responses. In particular, when the agent correctly matches the target card in 10 consecutive trials, the sorting rule is automatically changed. The task ends after completing a maximum of 128 trials.

 
___________________________________________________________________________________ 
Algorithm 1 Bayesian cognitive agent 
___________________________________________________________________________________ 
  1:  Set parameters θ = (λ,δ). 
  2:  Set initial activation levels ω0 = (0.5,0.5,0.5). 
  3:  Set initial observation x0 = ∅ and p(s1|x0) = p(s1). 
  4:  for t = 1,...,T do 
  5:       Sample feature from prior/predictive internal model st ∼ p(st|x0:t−1). 
  6:       Obtain a new observation xt = (at,ft). 
  7:       Compute state posterior p(st|x0:t). 
  8:       Compute new activation levels ωt. 
  9:       Compute stability matrix Γ(t). 
 10:        Update prior/predictive internal model to p(st+1|x0:t). 
 11:  end for 
____________________________________________________________________________________    

Simulation 1: clinical assessment of the Bayesian agent

Ideally, the qualitative performance of the Bayesian cognitive agent will resemble human performance. To this aim, we adopt a metric which is usually employed in clinical assessment of test results in neurological and psychiatric patients (Braff et al., 1991; Zakzanis, 1998; Bechara & Damasio, 2002; Landry & Al-Taie, 2016). Thus, agent performance is codified according to a neuropsychological criterion (Heaton, 1981; Flashman, Homer & Freides, 1991) which allows to classify responses into several response types. These response types provide the scoring measures for the test.

Table 1:

Descriptive summary of all quantities involved in our model representation.

Expression	Name	Description
st ∈ {1, 2, 3}	Sorting rule	Card feature relevant for the sorting criterion in trial t.
at ∈ {1, 2, 3, 4}	Choice action	Action of choosing one of the four stimulus cards in trial t.
ft ∈ {0, 1}	Feedback	Indicates whether the action of matching a stimulus to a target card is correct or not in trial t.
xt = (at, ft)	Observation	Pair of action and feedback which constitutes the agent’s observation in trial t.
Γ(t)	Stability matrix	Matrix encoding the agent’s beliefs about state transitions from trial t to the next trial t + 1.
λ ∈ [0, 1]	Flexibility	Parameter encoding the efficiency to disengage attention from a currently attended hidden state when signaled by the environment.
δ ∈ [0, 1]	Information loss	Parameter encoding how efficiently the agent’s internal model converges to the true environmental model based on experience.
$m_t^{\left(i\right)}\in \left\{0,1\right\}$	Matching signal	Signal indicating whether feature i is relevant in trial t based on the feedback received.
${\omega }_t^{\left(i\right)}\in \left[0,1\right]$	State activation level	Agent’s internal measure of the accrued evidence for the hidden environmental state i in trial t.
ℬt ∈ ℝ+	Bayesian surprise	Kullback–Leibler divergence between prior and posterior beliefs about hidden environmental states in trial t.
ℐt ∈ ℝ+	Shannon surprise	Information-theoretic surprise encoding the improbability or unexpectedness of an observation in trial t.
ℋt ∈ ℝ+	Entropy	Degree of epistemic uncertainty in the internal model of the environment in trial t.

DOI: 10.7717/peerj.10316/table-1

Here, we are interested in: (1) non-perseverative errors (E); (2) perseverative errors (PE); (3) number of trials to complete the first category (TFC); and (4) number of failures to maintain set (FMS). Perseverative errors occur when the agent applies a sorting rule which was valid before the rule has been changed. Usually, detecting a perseveration error is far from trivial, since several response configurations could be observed when individuals are required to shift a sorting rule after completing a category (see Flashman, Homer & Freides (1991) for details). On the other hand, non-perseverative errors refer to all errors which do not fit the above description, or in other words, do not occur as a function of changing the sorting rule, such as casual errors.

The number of trials to complete the first category tells us how many trials the agent needs in order to achieve the first sorting principle, and can be seen as an index of conceptual ability (Anderson, 2008; Singh, Aich & Bhattarai, 2017). Finally, a failure to maintain a set occurs when the agent fails to match cards according to the sorting rule after it can be determined that the agent has acquired the rule. A given sorting rule is assumed to be acquired when the individual correctly sorts at least five cards in a row (Heaton, 1981; Figueroa & Youmans, 2013). Thus, a failure to maintain a set arises whenever a participant suddenly changes the sorting strategy in the absence of negative feedback. Failures to maintain a set are mostly attributed to distractibility. We compute this measure by counting the occurrences of first errors after the acquisition of a rule.

We run the generative model by varying flexibility across four levels, λ ∈ {0.3, 0.5, 0.7, 0.9}, and information loss across three levels, δ ∈ {0.4, 0.7, 0.9}. We generate data from 150 synthetic cognitive agents per parameter combination and compute standard scoring measures for each of the agents simulated responses. Results from the simulation runs are depicted in Table 2 and a graphical representation is provided in Fig. 3.

The simulated performance of our Bayesian cognitive agents demonstrates that different parameter combinations capture different meaningful behavioral patterns. In other words, flexibility and information loss seem to interact in a theoretically meaningful way.

First, overall errors increase when flexibility (λ) decreases, which is reflected by the inverse relation between the number of casual, as well as perseverative, errors and the values of parameter λ. Moreover, this pattern is consistent across all the levels of parameter δ. More precisely, information loss (δ) seems to contribute to the characterization of the casual and the perseverative components of the error in a different way. Perseverative errors are likely to occur after a sorting rule has changed and reflect the inability of the agent to use feedback to disengage attention from the currently attended feature. They therefore result from local cognitive dynamics conditioned on a particular stage of the task (e.g., after completing a series of correct responses).

Table 2:

Mean clinical scoring measures as functions of flexibility (λ) and information loss (δ). Cells show the average scores across simulated agents (standard deviation is shown in parenthesis).

Scoring measure	Info. Loss (δ)	Flexibility (λ)
		λ = 0.3	λ = 0.5	λ = 0.7	λ = 0.9
Casual Errors (E)	δ = 0.4	9.07 (2.68)	7.95 (2.07)	7.50 (2.13)	6.85 (1.75)
	δ = 0.7	10.84 (2.35)	9.60 (2.2)	8.25 (2.23)	7.37 (1,74)
	δ = 0.9	12.75 (2.96)	11.25 (2.43)	9.12 (2.09)	7.79 (1.73)
Perseverative Errors (PE)	δ = 0.4	20.81 (2.27)	18.18 (1.88)	14.99 (1.88)	12.37 (1.12)
	δ = 0.7	19.77 (2.55)	17.65 (2.26)	15.42 (1.94)	12.39 (1.47)
	δ = 0.9	18.56 (2.76)	16.58 (2.53)	14.49 (2.03)	12.33 (1.44)
Trials to First Category (TFC)	δ = 0.4	12.20 (1.46)	11.91 (1.35)	11.83 (1.24)	11.67 (1.04)
	δ = 0.7	13.82 (2.76)	13.32 (2.52)	12.97 (2.13)	12.29 (1.53)
	δ = 0.9	17.27 (4.21)	16.63 (4.04)	14.39 (3.58)	12.91 (1.91)
Failures to Maintain Set (FMS)	δ = 0.4	0.11 (0.31)	0.09 (0.31)	0.05 (0.32)	0.02 (0.14)
	δ = 0.7	1.65 (1.4)	1.41 (1.3)	0.84 (0.91)	0.35 (0.69)
	δ = 0.9	4.44 (1.96)	3.88 (1.86)	2.79 (1.56)	1.54 (1.25)

DOI: 10.7717/peerj.10316/table-2

Second, information loss does not interact with flexibility when perseverative errors are considered. This is due to the fact that high information loss affects general performance by yielding a dysfunctional response strategy which increases the probability of making an error at any stage of the task. The lack of such interaction provides evidence that our computational model can disentangle between error patterns due to perseveration and those due to general distractibility, according to neuropsychological scoring criteria.

However, in our framework, flexibility (λ) is allowed to yield more general and non-local cognitive dynamics as well. Indeed, λ plays a role whenever belief updating is demanded as a function of negative feedback. An error classified as non-perseverative (e.g., casual error) by the scoring criteria might still be processed as a feedback-related evidence for belief updating. Consistently, the interaction between λ and δ in accounting for causal errors shows that performance worsens when both flexibility and information loss become less optimal, and that such pattern becomes more pronounced for lower values of δ.

On the other hand, a specific effect of information loss (δ) can be observed for the scoring measures related to slow information processing and distractibility. The number of trials to achieve the first category reflects the efficiency of the agent in arriving at the first true environmental model. Flexibility does not contribute meaningfully to the accumulation of errors before completing the first category for some levels of information loss. This is reflected by the fact that the mean number of trials increases as a function of δ, and do not change across levels of λ for low and mid values of δ. A similar pattern applies for failures to maintain a set. Both scoring measures index a deceleration of the process of evidence accumulation for a specific environmental configuration, although the latter is a more exhaustive measures of dysfunctional adaptation.

Therefore, an interaction between parameters can be observed when information loss is high. A slow internal model convergence process increases the amount of errors due to improper rule sampling from the internal environmental model. However, internal model convergence also plays a role when a new category has to be accomplished after completing an older one. On the one hand, compromised flexibility increases the amount of errors due to inefficient feedback processing. This leads to longer trial windows needed to achieve the first category. On the other hand, when information loss is high, belief updating upon negative feedback is compromised due to high internal model uncertainty. At this point, the probability to err due to distractibility increases, as accounted by the failures to maintain a set measures.

Finally, the joint effect of δ and λ for high levels of information loss suggests that the roles played by the two cognitive parameters in accounting for adaptive functioning can be entangled when neuropsychological scoring criteria are considered.

Simulation 2: Information-theoretic analysis of the Bayesian agent

In the following, we explore a different simulation scenario in which information-theoretic measures are derived to assess performance of the Bayesian cognitive agent. In particular, we explore the functional relationship between cognitive parameters and the dynamics of the recovered information-theoretic measures by simulating observed responses by varying flexibility across three levels, λ ∈ {0.1, 0.5, 0.9}, and information loss across three levels, δ ∈ {0.1, 0.5, 0.9}.

For this simulation scenario, we make no prior assumptions about sub-types of error classification. Instead, we investigate the dynamic interplay between Bayesian surprise, ${\mathcal{B}}_t$, Shannon surprise, ${\mathcal{I}}_t$, and entropy, ${\mathcal{H}}_t$ over the entire course of 128 trials in the WCST.

Figure 4 depicts results from the nine simulation scenarios. Although an exhaustive discussion on cognitive dynamics should couple information-theoretic measures with patterns of correct and error responses, we focus solely on the information-theoretic time series for illustrative purposes. We refer to the ‘Application’ for a more detailed description of the relation between observed responses and estimated information-theoretic measures in the context of data from a real experiment.

Again, simulated performance of the Bayesian cognitive agent shows that different parameter combinations yield different patterns of cognitive dynamics. Observed spikes and their related magnitudes signal informative task events (e.g., unexpected negative feedback), as accounted by Shannon surprise, or belief updating, as accounted by Bayesian surprise. Finally, entropy encodes the epistemic uncertainty about the environmental model on a trial-by-trial basis.

In general, low information loss (δ) ensures optimal behavior by speeding up internal model convergence by decreasing the number of trials needed to minimize uncertainty about the environmental states. Low uncertainty reflects two main aspects of adaptive behavior. On the one hand, the probability that a response occurs due to sampling of improper rules decreases, allowing the agent to prevent random responses due to distractibility. On the other hand, model convergence entails a peaked Shannon surprise when a negative feedback occurs, due to the divergence between predicted and actual observations.

Flexibility (λ) plays a crucial role in integrating feedback information in order to enable belief updating. The first row depicted in Fig. 4 shows cognitive dynamics related to low information loss, across the levels of flexibility. As can be noticed, there is a positive relation between the magnitude of the Bayesian surprise and the level of flexibility, although unexpectedness yields approximately the same amount of signaling, as accounted by peaked Shannon surprise. From this perspective, surprise and belief updating can be considered functionally separable, where the first depends on the particular internal model probability configuration related to δ, whilst the second depends on flexibility λ.

However, more interesting patterns can be observed when information loss increases. In particular, model convergence slows down and several trials are needed to minimize predictive model entropy. Casual errors might occur within trial windows characterized by high uncertainty, and interactions between entropy and Shannon surprise can be observes in such cases. In particular, Shannon surprise magnitude increases when model’s entropy decreases, that is, during task phases in which the internal model has already converged. As a consequence, negative feedback could be classified as informative or uninformative, based on the uncertainty in the current internal model. This is reflected by the negative relation between entropy and Shannon surprise, as can be noticed by inspecting the graphs depicted in the third row of Fig. 4. Therefore, the magnitude of belief updating depends on the interplay between entropy and Shannon surprise, and can differ based on the values of the two measures in a particular task phase.

In sum, both simulation scenarios suggest that the simulated behavior of our generative model is in accord with theoretical expectations. Moreover, the flexibility and information loss parameters can account for a wide range of observed response patterns and inferred dynamics of information processing.

Parameter estimation

In this section, we discuss the computational framework for estimating the parameters of our model from observed behavioral data. Parameter estimation is essential to inferring the cognitive dynamics underlying observed behavior in real-world applications of the model. This section is slightly more technical and can be skipped without significantly affecting the flow of the text.

Rationale

The advantage of modeling cognitive dynamics in individuals from a clinical population is that model predictions can be examined in light of available evidence about individual performance. For instance, SDIs are known to demonstrate inefficient conceptualization of the task and dysfunctional, error-prone response strategies. This has been attributed to defective error monitoring and behavior modulation systems, which depend on cingulate and frontal brain regions functionality (Kübler, Murphy & Garavan, 2005; Willuhn, Sun & Steiner, 2003). On the other hand, the WCST should be a rather easy and straightforward task for healthy participants to obtain excellent performance. Therefore, we expect our model to consistently capture such characteristics. To test these expectations, we estimate the two relevant parameters λ and δ from both clinical patients and healthy controls from an already published dataset (Bechara & Damasio, 2002).

The data

The dataset used in this application consists of responses collected by administering the standard Heaton version of the WCST (Heaton, 1981) to healthy participants and SDIs. In this version of the task, the sorting rule changes when a participant collects a series of 10 consecutive correct responses, and the task ends when this happens for 6 times. Participants in the study consisted of 39 SDIs and 49 healthy individuals. All participants were adults (>18 years old) and gave their informed consent for inclusion which was approved by the appropriate human subject committee at the University of Iowa. SDIs were diagnosed as substance dependent based on the Structured Clinical Interview for DSM-IV criteria (First, 1997).

Model fitting

We fit the Bayesian cognitive agent separately to data from each participant in order to obtain individual-level posterior distributions. We apply the same BayesFlow network trained for the previous simulation studies, so obtaining posterior samples for each participant is almost instant (due to amortized inference).

Results

The means of the joint posterior distributions are depicted for each individual in Fig. 6, and provide a complete overview of the heterogeneity in cognitive sub-components at both individual and group levels (individual-level full joint posterior distributions can be found in the Appendix A1).

The estimates reveal a rather interesting pattern across both healthy and SDI participants. In particular, in both clinical and control groups, individuals with a poor flexibility (e.g., low values of λ) can be detected. However, the group parameter space appears to be partitioned into two main clusters consisting of individuals with high and low flexibility, respectively. As can be noticed, the majority of SDIs belongs to the latter cluster, which suggests that the model is able to capture error-related defective behavior in the clinical population and attribute it specifically to the flexibility parameter. On the other hand, individual performance seems hardly separable along the information loss parameter dimension.

As a further validation, we compare the classification performance of two logistic regression models. The first uses the estimated parameter means as inputs and the participants’ binary group assignment (patient vs. control) as an outcome. The second uses the four standard clinical measures (non-perseverative errors (E), perseverative errors (PE), number of trials to complete the first category (TFC), number of failures to maintain set (FMS) computed from the sample as inputs and the same outcome. Since we are interested solely in classification performance and want to mitigate potential overfitting due to small sample size, we compute leave-one-out cross-validated (LOO-CV) performance for both models. Interestingly, both logistic regression models achieve the same accuracy of 0.70, with a sensitivity of 0.71 and specificity of 0.70. Thus, it appears that our model is able to differentiate between SDIs and healthy individuals as good as the standard clinical measures.

However, as pointed out in the previous sections, estimated parameters serve merely as a basis to reconstruct cognitive dynamics by means of the trial-by-trial unfolding of information-theoretic measures. Moreover, cognitive dynamics can only be analysed and interpreted by relying on the joint contribution of both estimated parameters and individual-specific observed response patterns.

To further clarify this concept, we investigate the reconstructed time series of information-theoretic quantities based on the response patterns of two exemplary individuals (Fig. 7). In particular, Fig. 7A depicts the behavioral outcomes of a SDI with sub-optimal performance where the information-theoretic trajectories are reconstructed by taking the corresponding posterior means ($\left[\bar{\lambda }=0.07,\bar{\delta }=0.82\right]$), thus representing compromised flexibility and high information loss. Differently, Fig. 7B shows the information-theoretic path related to response dynamics of an optimal control participant, according to the parameter set $\left[\bar{\lambda }=0.60,\bar{\delta }=0.35\right]$, representing relatively high flexibility, and low information loss. Note, that in both cases, the reconstructed information-theoretic measures are based on the estimated posterior means for ease of comparison (see Appendix A1 for the full joint posterior densities of the two exemplary individuals and the rest of the sample).

Results in Fig. 7A account for a typical sub-optimal behavior observed in the SDI group, where several errors are produced in different phases of the task. The error patterns produced by such an individual might be induced by a non-trivial interaction between cognitive sub-components. Lower values of flexibility imply that errors are likely to be produced by generating responses from an internal environmental model which is no longer valid. In other words, the agent is unable to rely on local feedback-related information in order to update beliefs about hidden states. On the other hand, higher values of information loss reflect a general inefficiency of belief updating processes due to slow convergence to the optimal probabilistic environmental model. From this perspective, Bayesian surprise ${\mathcal{B}}_t$ and Shannon surprise ${\mathcal{I}}_t$ might play different roles in regulating behavior based on different internal model probability configurations. In addition, errors might be processed differently based on the status of the internal environmental representation, as reflected by the entropy of the predictive model, ${\mathcal{H}}_t$. Thus, information-theoretic measures allow to describe cognitive dynamics on a trial-by-trial basis and, further, to disentangle the effect that different feedback-related information processing dynamics exert on adaptive behavior.

Processing unexpected observations is accounted by the quantification of surprise upon observing a response-feedback pair which is inconsistent with the current internal model of the task environment. Negative feedback is maximally informative when errors occur after the internal model has converged to the true task model (grey area, Fig. 7A), or the entropy approaches zero (grey line, Fig. 7A). The Shannon surprise (orange line) is maximal when errors occur within trial windows in which the agent’s uncertainty about environmental states is minimal (orange areas, Fig. 7A). However, internal model updates following an informative feedback are not optimally performed, which is reflected by very small Bayesian surprise (blue line, Fig. 7A). This can be attributed to impaired flexibility and reflects the fact that after internal model convergence, informative feedback is not processed adequately and the internal model becomes impervious to change.

Conversely, errors occurring when the agent is uncertain about the true environmental state carry no useful information for belief updating, since the system fails to conceive such errors as unexpected and informative. The information loss parameter plays a crucial role in characterizing this cognitive behavior. The slow convergence to the true environmental model, accompanied by the slow reduction of entropy in the predictive model, leads to a large number of trials required to achieve a good representation of the current task environment (white areas, Fig. 7A). Errors occurring within trial windows with large predictive model entropy (green area, Fig. 7A) do not affect subsequent behavior, and feedback is maximally uninformative.

Rather different cognitive dynamics can be observed in Fig. 7B, accounting for a typical optimal behavior where the errors produced fall within the trial windows which follow a rule completion (e.g., when the individual completes a sequence of 10 consecutive correct responses), and, thus, the environmental model becomes obsolete. However, the high flexibility, λ, allows to rely on local feedback-related information to suddenly update beliefs about the hidden states, that is, the most appropriate sorting rule. In this case, negative feedback become maximally informative after model convergence (grey area, Fig. 7B) and the process of entropy reduction (green line, Fig. 7B) is faster (e.g., less trials are needed) compared to the sub-optimal behavior scenario. Since uncertainty about the environmental states decreases faster, the Shannon surprise is always highly peaked when errors occur (orange line, Fig. 7B), thus ensuring an efficient employment of the local feedback-related information. Accordingly, higher values of Bayesian surprise are observed (blue line, Fig. 7B), revealing optimal internal model updating.

In general, the role that predictive (internal) model uncertainty plays in characterizing the way the agent processes feedback allows to disentangle sub-types of errors based on the information they convey for subsequent belief updating. From this perspective, error classification is entirely dependent on the status of the internal environmental model across task phases. Identifying such a dynamic latent process is therefore fundamental, since the error codification criterion evolves with respect to the internal information processing dynamics. Otherwise, the problem of inferring which errors are due to perseverance in maintaining an older (converged) internal model and which due to uncertainty about the true environmental state becomes intractable, or even impossible.