ABSTRACT

During the development of large-scale school achievement tests, panels of independent subject-matter experts use systematic judgmental methods to rate the correspondence between a given test’s items and performance objective statements. The individual experts’ ratings may then be used to compute summary indices to quantify the match between a given test and its target item domain. The magnitude of alignment index variability across experts within a panel, and randomly-sampled panels, is largely unknown, however. Using rater-by-item data from alignment reviews of 14 US states’ achievement tests, we examine observed distributions and estimate standard errors for three alignment indices developed by Webb. Our results suggest that alignment decisions based on the recommended criterion for the balance-of-representation index may often be uncertain, and that the criterion for the depth-of-knowledge consistency index should perhaps be reconsidered. We also examine current recommendations about the number of expert panelists required to compute these alignment indices.

摘要

在大规模学校成绩测试的开发过程中，独立主题专家小组使用系统的判断方法来评估给定测试项目与绩效目标陈述之间的对应关系。然后，可以使用各个专家的评分来计算摘要索引，以量化给定测试与其目标项目域之间的匹配。然而，专家组和随机抽样的专家组之间的比对指数变异性的大小在很大程度上尚不清楚。我们使用美国14个州的成就测验的一致性审核中的按项目进行评分的数据，我们检查了观察到的分布并估算了由Webb开发的三个一致性指标的标准误差。我们的结果表明，根据代表平衡指标的推荐标准进行的一致性决策通常可能不确定，也许应该重新考虑知识深度一致性指数的标准。我们还将检查有关计算这些比对指数所需的专家小组成员人数的最新建议。