Abstract

The objective of the present study was to investigate item-, examinee-, and country-level correlates of rapid guessing (RG) in the context of the 2018 PISA science assessment. Analyzing data from 267,148 examinees across 71 countries showed that over 50% of examinees engaged in RG on an average proportion of one in 10 items. Descriptive differences were noted between countries on the mean number of RG responses per examinee with discrepancies as large as 500%. Country-level differences in the odds of engaging in RG were associated with mean performance and regional membership. Furthermore, based on a two-level cross-classified hierarchical linear model, both item- and examinee-level correlates were found to moderate the likelihood of RG. Specifically, the inclusion of items with multimedia content was associated with a decrease in RG. A number of demographic and attitudinal examinee-level variables were also significant moderators, including sex, linguistic background, SES, and self-rated reading comprehension, motivation mastery, and fear of failure. The findings from this study imply that select subgroup comparisons within and across nations may be biased by differential test-taking effort. To mitigate RG in international assessments, future test developers may look to leverage technology-enhanced items.

摘要

本研究的目的是在 2018 年 PISA 科学评估的背景下调查快速猜测 (RG) 的项目级、考生级和国家级相关性。分析来自 71 个国家/地区的 267,148 名考生的数据显示，超过 50% 的考生从事 RG 的平均比例为十分之一。各国之间在每个考生的 RG 回答的平均数量上存在描述性差异，差异高达 500%。参与 RG 几率的国家级差异与平均绩效和区域成员有关。此外，基于两级交叉分类层次​​线性模型，项目级和考生级相关性都被发现可以缓和 RG 的可能性。具体来说，包含多媒体内容的项目与 RG 的减少有关。许多人口统计和态度的考生水平变量也是重要的调节因素，包括性别、语言背景、SES、自评阅读理解、动机掌握和对失败的恐惧。这项研究的结果表明，国家内部和国家之间的选择亚组比较可能会因不同的应试努力而产生偏差。为了减轻国际评估中的 RG，未来的测试开发人员可能会寻求利用技术增强的项目。