For results from large-scale surveys to inform policy and practice appropriately, all participants must interpret and respond to items similarly. While organizers of surveys assessing student outcomes often ensure this for achievement measures, doing so for psychological questionnaires is also critical. We demonstrate this by examining the dimensionality of reading self-concept—a crucial psychological construct for several outcomes—across reading achievement levels. We use Programme for International Student Assessment 2018 data (N = 529,966) and local structural equation models (LSEMs) to do so. Results reveal that reading self-concept dimensions (assessed through reading competence and difficulty) vary across reading achievement levels. Students with low reading achievement show differentiated responses to the two item sets (high competence–high difficulty). In contrast, students with high reading achievement have reconciled responses (high competence–low difficulty). Our results highlight the value of LSEMs in examining factor structure generalizability of constructs in large-scale surveys and call for greater cognitive testing during item development.

为了使大规模调查的结果适当地为政策和实践提供信息，所有参与者都必须以类似的方式解释和回应项目。虽然评估学生成绩的调查的组织者通常会确保这一点用于成绩衡量，但对心理问卷这样做也很重要。我们通过检查阅读自我概念的维度来证明这一点——阅读自我概念的一个重要心理结构——跨越阅读成就水平。我们使用 Program for International Student Assessment 2018 数据 ( N= 529,966）和局部结构方程模型（LSEM）来做到这一点。结果表明，阅读自我概念维度（通过阅读能力和难度评估）因阅读成绩水平而异。阅读成绩低的学生对两个项目集（高能力-高难度）表现出不同的反应。相比之下，阅读成绩高的学生有协调的反应（高能力-低难度）。我们的结果强调了 LSEM 在检查大规模调查中结构的因子结构普遍性方面的价值，并呼吁在项目开发过程中进行更多的认知测试。