Though much effort is often put into designing psychological studies, the measurement model and scoring approach employed are often an afterthought, especially when short survey scales are used (Flake & Fried, 2020). One possible reason that measurement gets downplayed is that there is generally little understanding of how calibration/scoring approaches could impact common estimands of interest, including treatment effect estimates, beyond random noise due to measurement error. Another possible reason is that the process of scoring is complicated, involving selecting a suitable measurement model, calibrating its parameters, then deciding how to generate a score, all steps that occur before the score is even used to examine the desired psychological phenomenon. In this study, we provide three motivating examples where surveys are used to understand individuals’ underlying social emotional and/or personality constructs to demonstrate the potential consequences of measurement/scoring decisions. These examples also mean we can walk through the different measurement decision stages and, hopefully, begin to demystify them. As we show in our analyses, the decisions researchers make about how to calibrate and score the survey used has consequences that are often overlooked, with likely implications both for conclusions drawn from individual psychological studies and replications of studies.

中文翻译：

---

调查评分决策如何影响您的研究结果：IRT 镜子之旅。


尽管设计心理学研究时经常投入大量精力，但所采用的测量模型和评分方法往往是事后才想到的，特别是在使用短调查量表时（Flake & Fried，2020）。测量被低估的一个可能原因是，除了由于测量误差导致的随机噪声之外，人们通常很少了解校准/评分方法如何影响常见的感兴趣估计值，包括治疗效果估计。另一个可能的原因是评分的过程很复杂，包括选择合适的测量模型，校准其参数，然后决定如何生成分数，所有在评分之前发生的步骤甚至用于检查所需的心理现象。在本研究中，我们提供了三个激励示例，其中调查用于了解个人潜在的社会情感和/或个性结构，以证明测量/评分决策的潜在后果。这些例子还意味着我们可以逐步了解不同的测量决策阶段，并希望能够开始揭开它们的神秘面纱。正如我们在分析中所示，研究人员做出的关于如何校准和评分所使用的调查的决定会产生经常被忽视的后果，这可能会对个人心理学研究和研究重复得出的结论产生影响。