ABSTRACT

The objective of this study was to evaluate whether differential noneffortful responding (identified via response latencies) was present in four countries administered a low-stakes college-level critical thinking assessment. Results indicated significant differences (as large as .90 SD) between nearly all country pairings in the average number of noneffortful responses per test taker. Furthermore, noneffortful responding was found to be associated with a number of individual-level predictors, such as demographics (both gender and academic year), prior ability, and perceived difficulty of the test, though, these predictors were found to differ across countries. Ignoring the presence of noneffortful responses was associated with: (a) model fit deterioration as well as inflation of reliability, and (b) the inclusion of non-invariant items in the score linking anchor set. However, no meaningful differences in relative performance were noted once accounting for noneffortful responses. Implications for test development and improving the validity of score-based inferences from international assessments are discussed.

摘要

这项研究的目的是评估在进行低风险大学水平批判性思维评估的四个国家中是否存在不同的不努力反应（通过反应潜伏期确定）。结果表明存在显着差异（高达0.90 SD）之间的差额，几乎每个国家/地区的配对中，每位应试者做出的不努力回答的平均数。此外，发现不适当的回答与许多个人层面的预测因素有关，例如人口统计学（性别和学年），先验能力和测试的感知难度，尽管这些预测因素在不同国家之间有所不同。忽略不妥协的存在与以下因素有关：（a）模型拟合恶化以及可靠性膨胀，以及（b）在评分链接锚集中包含非不变项。但是，一旦考虑到不正确的回答，就没有发现相对绩效方面的有意义的差异。讨论了对测试开发和提高国际评估中基于分数的推断的有效性的影响。