Purpose This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact of raters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores. Methods Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and the Pearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whether correct answers were provided or not and differences between internists and other specialists using the t-test. Results The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answers were provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists. Conclusion Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore, raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting.

中文翻译：

---

Ebel标准制定方法在2019年春季实施的加拿大皇家内科医生与外科医生内科认证考试包括多项选择题

目的这项研究旨在评估2019年春季加拿大皇家内科医生与外科医生内科认证考试中由多项选择题组成的Ebel标准制定方法的性能。具体而言，评估了以下参数：评估者之间的一致性，Ebel得分与项目设施指数之间的相关性，评估者正确答案知识对Ebel得分的影响以及评估者的专业知识对评估者之间的一致性和玉宝分数。方法数据来自加拿大皇家内科医生与外科医生学院认证考试。Ebel方法被49个评估者应用于203个多项选择题。设施指数来自194个候选人。我们计算了Ebel得分与项目设施指数之间的Fleiss kappa和Pearson相关性。我们根据是否提供正确答案以及内科医生与其他专家之间使用t检验的差异调查了Ebel分数的差异。结果设施和相关性的Fleiss kappa均低于0.15。提供正确答案时，Ebel分数与设施指数之间的相关性较低，而没有提供答案时，则可以忽略不计。无论是否提供正确答案，Ebel分数都是相同的。实习者和其他专家之间的评分者间一致性和Ebel得分没有显着差异。结论评分者之间的一致性以及项目Ebel得分与设施指数之间的相关性一直很低。此外，评估者对正确答案的了解和评估者的专业对目前的Ebel分数没有影响。