ABSTRACT

We report on a standard-setting project in which the Item-Descriptor-Matching Method (IDM) and a complementary benchmarking approach were employed to align a suite of English language proficiency exams to the Common European Framework of Reference (CEFR), with a particular focus on the integrated and independent writing exams. Judges’ ratings on eight writing tasks and 48 test taker scripts were collected online via SmartSurvey. The judges gave CEFR-level jkudgements for tasks and scripts, they stated which CEFR descriptors they matched against the task demands and scripts, and they evaluated the combined approach and outcomes. Thus, it was possible to monitor how judges applied and interpreted the CEFR descriptors, a prerequisite for establishing alignment validity. Analyses of judgement consistency revealed a high level of consistency on the task judgements and overall performance ratings, yet they also revealed some variations in the selected CEFR descriptors underpinning the judgements. Making these variations transparent facilitated a targeted discussion with an explicit focus on the CEFR, i.e. the framework to which the tests were to be aligned. Overall, the judges reported confidence in using the combined approaches, in their judgements and in the recommended CEFR cut-scores, thus corroborating procedural validity.

摘要

我们报告了一个标准制定项目，在该项目中，项目描述符匹配方法（IDM）和补充基准测试方法被用来使一套英语水平考试与欧洲共同参考标准保持一致（CEFR），特别侧重于综合和独立写作考试。通过SmartSurvey在线收集了法官对八项写作任务和48种应试者脚本的评分。评委们对任务和脚本进行了CEFR级别的判断，指出了与任务需求和脚本相匹配的CEFR描述符，并评估了组合方法和结果。因此，可以监视法官如何应用和解释CEFR描述符，这是建立比对有效性的前提。判断一致性的分析显示了任务判断和整体绩效等级的高度一致性，但同时也揭示了所选CEFR描述符在判断基础上的某些差异。使这些变化透明化有助于进行有针对性的讨论，明确集中于CEFR，即与测试相适应的框架。总体而言，法官报告说他们有信心使用组合方法，他们的判断和推荐的CEFR评分标准，从而证实了程序的有效性。