Asymptotically Corrected Person Fit Statistics for Multidimensional Constructs with Simple Structure and Mixed Item Types

Abstract

Person fit statistics are frequently used to detect aberrant behavior when assuming an item response model generated the data. A common statistic, $l_z$, has been shown in previous studies to perform well under a myriad of conditions. However, it is well-known that $l_z$ does not follow a standard normal distribution when using an estimated latent trait. As a result, corrections of $l_z$, called $l_z^*$, have been proposed in the literature for specific item response models. We propose a more general correction that is applicable to many types of data, namely survey or tests with multiple item types and underlying latent constructs, which subsumes previous work done by others. In addition, we provide corrections for multiple estimators of $\theta$, the latent trait, including MLE, MAP and WLE. We provide analytical derivations that justifies our proposed correction, as well as simulation studies to examine the performance of the proposed correction with finite test lengths. An applied example is also provided to demonstrate proof of concept. We conclude with recommendations for practitioners when the asymptotic correction works well under different conditions and also future directions.

Appendix A

1.1 Formulas for Different Estimators of $\varvec{\theta }$: $\varvec{\theta }_{MAP}$ and $\varvec{\theta }_{WLE}$

The following section is based on work done by Sinharay (2016a) and Wang (2015). Suppose $\varvec{\theta }$ is estimated by $\varvec{{\hat{\theta }}}$, where $\varvec{{\hat{\theta }}}$ satisfies the following condition:

$$\begin{aligned} \nabla l(X|{\varvec{\theta }})|_{\varvec{\theta }} = \begin{bmatrix} t_{01}(\varvec{\theta }) \\ t_{02}(\varvec{\theta }) \\ \vdots \\ t_{0S}(\varvec{\theta }) \end{bmatrix} + \begin{bmatrix} \sum _{i=1}^{p} \sum _{j=0}^{m_i} \Big ( \mathbb {I}_j(X_{i}) - P_{ij}(\varvec{\theta })\Big ) t_{ij1}(\varvec{\theta }) \\ \sum _{i=1}^{p} \sum _{j=0}^{m_i} \Big ( \mathbb {I}_j(X_{i}) - P_{ij}(\varvec{\theta })\Big ) t_{ij2}(\varvec{\theta }) \\ \vdots \\ \sum _{i=1}^{p} \sum _{j=0}^{m_i} \Big ( \mathbb {I}_j(X_{i}) - P_{ij}(\varvec{\theta })\Big ) t_{ijS}(\varvec{\theta }) \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \end{aligned}$$

(63)

which can be rewritten as:

$$\begin{aligned} \nabla l(X|{\varvec{\theta }})|_{\varvec{\theta }} =t_{0}(\varvec{\theta }) + \sum _{i=1}^{p} \sum _{j=0}^{m_i} \Big ( \mathbb {I}_j(X_{i}) - P_{ij}(\varvec{\theta })\Big ) t_{ij}(\varvec{\theta }) = \mathbf {0} \end{aligned}$$

(64)

for some functions $t_{0}(\varvec{\theta })= (t_{01}, t_{02}, \ldots , t_{0S})'$ and $t_{ij}(\varvec{\theta }) = (t_{ij1}, t_{ij2}, \ldots , t_{ijS})'$. For instance, $\hat{\varvec{\theta }}_{ML}$ is the value of $\varvec{\theta }$ for which:

$$\begin{aligned} \nabla l(X|{\varvec{\theta }})|_{\varvec{\theta }} = \sum _{i=1}^{p} \sum _{j=0}^{m_i} \Big ( \mathbb {I}_j(X_{i}) - P_{ij}(\varvec{\theta })\Big ) P_{ij}^{-1}({\varvec{\theta }}) \nabla P_{ij}({\varvec{\theta }})|_{\varvec{\theta }} = \mathbf {0}. \end{aligned}$$

(65)

The equality in Equation (64) holds where $t_{0}(\varvec{\theta })$ and $t_{ij}(\varvec{\theta })$ satisfy:

$$\begin{aligned} t_{0}(\varvec{\theta }) = \mathbf {0} \text{ and } t_{ij}(\varvec{\theta }) = P_{ij}^{-1}({\varvec{\theta }}) \nabla P_{ij}({\varvec{\theta }})|_{\varvec{\theta }}. \end{aligned}$$

(66)

Similarly, $\varvec{{\hat{\theta }}}_{MAP}$ satisfies the following:

$$\begin{aligned} \nabla l(X|{\varvec{\theta }})|_{\varvec{\theta }} + \nabla \log \pi (\varvec{\theta })|_{\varvec{\theta }} = \mathbf {0}, \end{aligned}$$

(67)

where $\pi (\varvec{\theta })$ is a prior distribution for $\varvec{\theta }$. Equation (64) holds for $\varvec{{\hat{\theta }}}_{MAP}$ where:

$$\begin{aligned} t_{0}(\varvec{\theta }) = \nabla \log \pi (\varvec{\theta })|_{\varvec{\theta }} \text{ and } t_{ij}(\varvec{\theta }) = P_{ij}^{-1}({\varvec{\theta }}) \nabla P_{ij}({\varvec{\theta }})|_{\varvec{\theta }}. \end{aligned}$$

(68)

Note that if the prior is a standard multivariate normal distribution, then $\nabla \log \pi (\varvec{\theta })|_{\varvec{\theta }} = -\varvec{\theta }$. $\varvec{{\hat{\theta }}}_{WLE}$ satisfies Equation (64) where:

$$\begin{aligned} \nabla l(X|{\varvec{\theta }})|_{\varvec{\theta }} + \nabla \bar{\mathbf {I}}_p(\varvec{\theta })\mathbf {B}(\varvec{\theta })|_{\varvec{\theta }} = \mathbf {0}, \end{aligned}$$

(69)

where $\bar{\mathbf {I}}_p$ be the average information about $\varvec{\theta }$ in the sample where $\bar{\mathbf {I}}_p = \sum _{i=1}^p \mathbf {I}_i(\varvec{\theta })/p$. $\mathbf {B}(\varvec{\theta }) = [B(\theta _1), B(\theta _2), ..., B(\theta _S)]'$ is a S-dimensional vector where the $s^{th}$ element in $\mathbf {B}(\varvec{\theta })$ is:

$$\begin{aligned} \frac{1}{2}\sum _{t,u,v=1}^p I^{uv}I^{vt}E\Big ( \frac{\partial ^3l}{\partial \theta _t \partial \theta _u\partial \theta _v}\Big ) \end{aligned}$$

(70)

Therefore this satisfies Equation (64) where

$$\begin{aligned} t_{0}(\varvec{\theta }) = \nabla \bar{\mathbf {I}}_p(\varvec{\theta })\mathbf {B}(\varvec{\theta })|_{\varvec{\theta }} \text{ and } t_{ij}(\varvec{\theta }) = P_{ij}^{-1}({\varvec{\theta }}) \nabla P_{ij}({\varvec{\theta }})|_{\varvec{\theta }}. \end{aligned}$$

(71)