Setting cut scores on multistage-adaptive tests (MSTs) is difficult, particularly when the test spans several grade levels, and the selection of items from MST panels must reflect the operational test specifications. In this study, we describe, illustrate, and evaluate three methods for mapping panelists’ Angoff ratings into cut scores on the scale underlying an MST. The results suggest the test characteristic function and item characteristic curve methods performed similarly, but the method based on dichotomizing panelists’ ratings at a response probability of .67 was unacceptable. The study featured a rating booklet design that allowed us to systematically evaluate the validity of the Angoff ratings across test levels, which contributed internal validity evidence for the cut scores, which were also evaluated using procedural and external validity evidence. The implications of the results for future standard setting studies and research in this area are discussed.

中文翻译：

---

在多阶段自适应测试中设置和验证多个标准

在多阶段自适应测试 (MST) 上设置分数是很困难的，特别是当测试跨越多个年级时，并且从 MST 面板中选择的项目必须反映操作测试规范。在这项研究中，我们描述、说明和评估了三种方法，用于将小组成员的 Angoff 评级映射到 MST 基础量表上的削减分数。结果表明测试特征函数和项目特征曲线方法的表现相似，但基于响应概率为 0.67 的二分小组成员评级的方法是不可接受的。该研究采用评分手册设计，使我们能够系统地评估跨测试级别的 Angoff 评分的有效性，这为削减分数提供了内部有效性证据，还使用程序和外部有效性证据进行了评估。讨论了该结果对该领域未来标准制定研究和研究的影响。