Since the No Child Left Behind Act (No Child Left Behind [NCLB], 2001) was enacted, the Bookmark method has been used in many state standard setting studies (Karantonis and Sireci; Zieky, Perie, and Livingston). The purpose of the current study is to evaluate the criticism that when panelists are presented with data during the Bookmark standard setting process, these data are often misunderstood. We collected survey responses from eight panels of teachers who worked on an alternate assessment standard setting workshop. We found that although many panelists understood these data, others misunderstood them. For example, when panelists reviewed panel judgment statistics, some extrapolated beyond what these data represent. Our results include themes describing the types of misconceptions we observed, and the need for training and evaluation related to understanding and use of data used in standard setting. We share some suggestions for consideration when implementing the Bookmark method.

中文翻译：

---

评估小组成员对标准设定数据的理解

自《不让一个孩子掉队法案》（《不让一个孩子掉队》[NCLB]，2001 年）颁布以来，书签方法已被用于许多州标准制定研究（Karantonis 和 Sireci；Zieky、Perie 和 Livingston）。当前研究的目的是评估在书签标准制定过程中向小组成员提供数据时，这些数据经常被误解的批评。我们收集了八位参与替代评估标准制定研讨会的教师小组的调查回复。我们发现，尽管许多小组成员理解这些数据，但其他人却误解了它们。例如，当小组成员审查小组判断统计数据时，有些人超出了这些数据所代表的范围。我们的结果包括描述我们观察到的误解类型的主题，以及与理解和使用标准制定中使用的数据相关的培训和评估的需要。我们分享了一些建议，以供在实施 Bookmark 方法时考虑。