A Correct Response Model in knowledge structure theory

Highlights

• The Correct Response Model predicts the probability of a correct answer to any item

• It is more straightforward than the classical Basic Local Independence Model

• We investigate its testability, identifiability and characterizability

Abstract

In knowledge space theory, the (latent) knowledge state of a student consists of the subset of test items that he masters in principle. Even at a given stage of apprenticeship, the student’s knowledge state may vary in a given collection of subsets. The collection of all possible states of all potential students forms a knowledge structure. In the modeling of student answer production, the probabilities governing the knowledge state are parameters. Moreover, for each item, two additional parameters capture the probabilities of careless errors and lucky guesses in the answers to the item. From all the latter parameters, the Correct Response Model (CRM) predicts the probability of a correct answer to any isolated item. From the same parameters, the Basic Local Independence Model (BLIM) predicts the probability of any possible pattern of correct responses. Here, a particular pattern records at a given time all the items to which a given student would provide correct answers. While general properties of the BLIM (such as identifiability of parameters) have been investigated, the simpler Correct Response Model still requires scrutiny. The present paper investigates the CRM as regards testability, identifiability and characterizability. It either provides explicit answers or points out serious difficulties garnered from various mathematical disciplines.

Introduction

A toy example serves here to introduce the questions investigated in the paper. Consider a domain $Q$ consisting of three items:

$$Q=\{a,b,c\}.$$

We view each item as an elementary piece in some corpus of knowledge, or equally well as an item belonging to an examination test about the corpus. Moreover, we identify the knowledge of a person in the domain as the collection of items he masters. Because of dependencies (which may be based on the logical organization of the corpus, or just statistically observed in a given population of students), not any subset of the domain can be a state of knowledge. Suppose there are only four (potential) knowledge states, constituting the collection

$$\mathcal{K}=\{\varnothing ,\{a,b\},\{a,c\},Q\}.$$

In principle, the correctness of the answer provided by a student to any item depends only on his knowledge state. However, variations are routinely observed in such answers. This observation motivates the introduction of stochastic effects governed by probability values. So, assume the knowledge state of a student varies in the collection $\mathcal{K}$ according to (unknown) probabilities assigned to the states and denoted as

$$\pi (\varnothing ),\pi \left(\{a,b\}\right),\pi \left(\{a,c\}\right),\pi \left(Q\right).$$

Moreover, when a student prepares an answer to an item in his actual knowledge state, he might by inattention provide a false answer; or to the contrary, for an item outside of his state, he could by chance provide a correct answer. We thus introduce probabilities ${\beta }_a$, ${\beta }_b$, ${\beta }_c$ for careless errors in answering the three items, as well as probabilities ${\eta }_a$, ${\eta }_b$, ${\eta }_c$ for lucky guesses in those answers.

A first model (implicit in Doignon & Falmagne, 1999, page 270) defines the probability $\tau \left(q\right)$ of a correct answer to any isolated item $q$; we call it the Correct Response Model (CRM). It first conditions the probability of a correct answer to item $q$ on the state of the student:

$$\tau \left(q\right)=\sum _{K\in \mathcal{K}}Pr(q\mid K)\cdot \pi \left(K\right).$$

Then, it specifies each conditional probability $Pr(q\mid K)$ by taking into account the careless error probabilities ${\beta }_q$ and the lucky guess probabilities ${\eta }_q$. In the case of the item $b$ in our toy example, the CRM sets

$$\tau \left(b\right)\stackrel{\text{def}}{=}(1-{\beta }_b)\cdot \pi \left(\{a,b\}\right)+(1-{\beta }_b)\cdot \pi \left(Q\right)+{\eta }_b\cdot \pi (\varnothing )+{\eta }_b\cdot \pi \left(\{a,c\}\right)=(1-{\beta }_b)\cdot \left(\pi \left(\{a,b\}\right)+\pi \left(Q\right)\right)+{\eta }_b\cdot \left(\pi (\varnothing )+\pi \left(\{a,c\}\right)\right).$$

In general, with the notations

$${\mathcal{K}}_q=\{K\in \mathcal{K}\mid q\in K\},{\mathcal{K}}_{\overline{q}}=\{K\in \mathcal{K}\mid q\notin K\},$$

the probability of a correct answer to the item $q$ equals

$$\tau \left(q\right)=(1-{\beta }_q)\cdot \pi \left({\mathcal{K}}_q\right)+{\eta }_q\cdot \pi \left({\mathcal{K}}_{\overline{q}}\right).$$

A second model, the Basic Local Independence Model (BLIM), is due to Falmagne and Doignon (1988a) (see also Doignon & Falmagne, 1999 and Falmagne & Doignon, 2011). It defines the probability of each response pattern, where a particular response pattern collects the items from the domain to which a student would provide a correct answer at a given time of his apprenticeship. For instance, $a$ correct, $b$ false, $c$ correct is summarized by the pattern $\{a$, $c\}$ (a subset of $Q$). According to the BLIM, the latter pattern is observed with a probability equal to

$$\begin{array}{ccc}\hfill \rho \left(\{a,c\}\right)& \hfill =\hfill & \sum _{K\in \mathcal{K}}Pr\left(\{a,c\}\mid K\right)\cdot \pi \left(K\right)\hfill \\ \hfill & \hfill \stackrel{\text{def}}{=}\hfill & \phantom{+}{\eta }_a(1-{\eta }_b){\eta }_c\cdot \pi (\varnothing )\hfill \\ \hfill & \hfill \hfill & +(1-{\beta }_a){\beta }_b{\eta }_c\cdot \pi \left(\{a,b\}\right)\hfill \\ \hfill & \hfill \hfill & +(1-{\beta }_a)(1-{\eta }_b)(1-{\beta }_c)\cdot \pi \left(\{a,c\}\right)\hfill \\ \hfill & \hfill \hfill & +(1-{\beta }_a){\beta }_b(1-{\beta }_c)\cdot \pi \left(Q\right).\hfill \end{array}$$

Both models rest on independence assumptions underlying their fundamental formulas in Eqs. (6), (7) (the formulation of the latter equation for a general pattern should be clear—anyway see later Eq. (9)). The models respectively define the probability of a correct response to each isolated item (CRM) or the probability of a whole pattern of responses (BLIM). Both rely on a fixed collection $\mathcal{K}$ of states (some specified subsets of a domain $Q$) together with the values of the various parameters: the probabilities $\pi \left(K\right)$ of the states $K$ in $\mathcal{K}$ and the probabilities ${\beta }_q$ of careless errors and ${\eta }_q$ of lucky guesses for the items $q$ in the domain $Q$. A particular case plays an important role in previous investigations (Falmagne and Doignon, 1988a, Falmagne and Doignon, 1988b), as well as here: by definition, the straight case obtains when ${\beta }_q={\eta }_q=0$ for all items $q$, that is, guessing and distraction are stochastically excluded.

Both the CRM and the BLIM are instances of probabilistic models. According to some formal process or formula, such a model predicts from any parameter point a distribution of probability values for some observables. Let us use the term predicted distribution or predicted point to abbreviate “any distribution of probability values for the observables that the model predicts”. Bamber and van Santen (2000) precisely state two fundamental questions about a probabilistic model, (i) model testability: is there any distribution of probability values for the observables that the model does not predict? (ii) model identifiability: is each predicted distribution produced from at most one parameter point? It is also common to investigate a third question, (iii) model characterizability: are the predicted distributions susceptible of an effective characterization (without reference to the underlying parameter values)?

Recently, several researchers have produced nice results about identifiability of the BLIM, the model for pattern probabilities. For instance, Stefanutti, Heller, Anselmi, and Robusto (2012) offer results on local identifiability of the model (this concept, as well as the related ones we need about models, will be precisely defined in Section 3).  Spoto, Stefanutti, and Vidotto (2013) construct families of collections $\mathcal{K}$ of states for which the BLIM is not identifiable.  Heller (2017) further investigates local identifiability and so-called trade-offs among parameters.  Stefanutti, Spoto, and Vidotto (2018) derive non-identifiability of the BLIM in case certain parameter transformations do exist. More recently, Stefanutti and Spoto (2020) further investigate groups of such transformations. Here, we consider the three fundamental questions for the first model, namely the correct response model (CRM). We reformulate (i) testability and (ii) identifiability, and investigate (iii) characterizability. We also point out serious difficulties for providing answers in the form of practically efficient criteria: we exhibit appropriate subcases which in other areas of mathematical research are deemed to be hopeless. Our results are often stated for the straight case. One of them indicates how to restrict in a natural way the parameter domain in order to make the modified CRM identifiable, at least for a large family of knowledge structures. Doignon, Heller, and Stefanutti (2018) is a survey chapter on the fundamental questions on probabilistic models, with the CRM and the BLIM as prominent examples. Here we give all proofs for results on the CRM, and we expose in a much simpler way how to restore identifiability.