ABSTRACT

Questionnaire scales that are mixed-worded, i.e. include both positively and negatively worded items, often suffer from issues like low reliability and more complex latent structures than intended. Part of the problem might be that some responders fail to respond consistently to the mixed-worded items. We investigated the prevalence and impact of inconsistent responders in 37 primary education systems participating in the joint PIRLS/TIMSS 2011 assessment. Using the mean absolute difference method and three mixed-worded self-concept scales, we identified between 2%‒36% of students as inconsistent responders across education systems. Consistent with expectations, these students showed lower average achievement scores and had a higher risk of being identified as inconsistent on more than one scale. We also found that the inconsistent responders biased the estimated dimensionality and reliability of the scales. The impact on external validity measures was limited and unsystematic. We discuss implications for the use and development of questionnaire scales.

摘要

混合措辞的问卷量表，即包括正面和负面措辞的项目，经常遭受诸如低可靠性和比预期更复杂的潜在结构等问题。部分问题可能是某些响应者未能始终如一地响应混合词项。我们调查了参与 PIRLS/TIMSS 2011 联合评估的 37 个小学教育系统中反应不一致的情况和影响。使用平均绝对差法和三个混合词的自我概念量表，我们确定了 2%-36% 的学生在整个教育系统中是不一致的响应者。与预期一致，这些学生的平均成绩分数较低，并且在多个量表上被认定为不一致的风险较高。我们还发现，不一致的响应者会影响量表的估计维度和可靠性。对外部有效性测量的影响是有限且不系统的。我们讨论了问卷量表的使用和发展的意义。