ABSTRACT

This paper details a two-stage process for standard setting using an extended pairwise comparison method. It describes the application of a new process of establishing a set of scaled and ordered performances as a basis for exemplifying standards. In stage one, the method of pairwise comparison was used to generate data from pairwise judgements made at the criterion level and overall grade. In stage two, judges identified where the pass/fail boundary lay in relation to a selected subset of ordered performances generated from stage one. The combined evidence from the two-stage process demonstrates the method’s application to establish exemplified standards using on-balance judgements across multiple criteria. The approach combines expert judgement and statistical data to provide a basis for comparable cross-institution assessments against standards in the case of a summative assessment of professional competence in teacher education. It may be relevant to any discipline with professional standards and complex performance assessments.

摘要

本文详细介绍了使用扩展的成对比较方法进行标准设置的两阶段过程。它描述了建立一组按比例缩放和有序性能的新过程的应用，以此作为例证标准的基础。在第一阶段，使用成对比较的方法从在标准水平和总体等级上做出的成对判断中生成数据。在第二阶段，法官确定与第一阶段生成的有序表演的选定子集相关的通过/失败边界在哪里。来自两阶段过程的综合证据表明，该方法适用于使用跨多个标准的平衡判断来建立示例性标准。这种方法结合了专家判断和统计数据，为教师教育专业能力的总结评估提供了依据标准进行跨机构评估的基础。它可能与具有专业标准和复杂绩效评估的任何学科有关。