Background:

OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs.

Purpose/aim:

The purpose of the study was comparing the utility, feasibility and appropriateness of 4 different standard setting methods with OSCEs at XUSOM.

Methods:

A 15-station OSCE was administered to 173 students over 6 months. Five stations were conducted for each organ system (Respiratory, Gastrointestinal and Cardiovascular). Students were assessed for their clinical skills in 15 stations. Four different standard setting methods were applied and compared with a control (Traditional method) to establish cut off scores for pass/fail decisions.

Results:

OSCE checklist scores revealed a Cronbach’s alpha of 0.711, demonstrating acceptable level of internal consistency. About 13 of 15 OSCE stations performed well with “Alpha if deleted values” lower that 0.711 emphasizing the reliability of OSCE stations. The traditional standard setting method (cut off score of 70) resulted in highest failure rate. The Modified Angoff Method and Relative methods yielded the lowest failure rates, which were typically less than 10% for each system. Failure rates for the Borderline methods ranged from 28% to 57% across systems.

Conclusions:

In our study, Modified Angoff method and Borderline regression method have shown to be consistently reliable and practically suitable to provide acceptable cut-off score across different organ system. Therefore, an average of Modified Angoff Method and Borderline Regression Method appeared to provide an acceptable cutoff score in OSCE. Further studies, in high-stake clinical examinations, utilizing larger number of judges and OSCE stations are recommended to reinforce the validity of combining multiple methods for standard setting.

中文翻译：

---

比较加勒比医学院客观结构化临床考试的标准制定方法

背景：

OSCE 广泛用于评估医学院的临床技能培训。使用传统的合格不合格截止值会在不同学生群体的结果中产生很大差异。这导致欧安组织越来越重视标准制定程序的应用。

目的/目标：

该研究的目的是比较 4 种不同的标准制定方法与 XUSOM 的 OSCE 的实用性、可行性和适当性。

方法：

在 6 个月的时间里，对 173 名学生进行了 15 站 OSCE。对每个器官系统（呼吸系统、胃肠道系统和心血管系统）进行了五个测站。在 15 个站点对学生的临床技能进行了评估。应用四种不同的标准设定方法并与对照（传统方法）进行比较，以确定通过/未通过决策的截止分数。

结果：

OSCE 检查表分数显示 Cronbach 的 alpha 值为 0.711，表明内部一致性可以接受。15 个 OSCE 站中约有 13 个表现良好，“Alpha if deleted values”低于 0.711，强调了 OSCE 站的可靠性。传统的标准设置方法（截止分数为 70）导致最高的失败率。改进的 Angoff 方法和相关方法产生了最低的故障率，每个系统的故障率通常低于 10%。跨系统的 Borderline 方法的失败率在 28% 到 57% 之间。

结论：

在我们的研究中，改进的 Angoff 方法和 Borderline 回归方法已被证明是一致可靠的，并且实际上适用于在不同器官系统中提供可接受的截止分数。因此，修正 Angoff 方法和边界回归方法的平均值似乎在 OSCE 中提供了可接受的截止分数。建议进一步研究，在高风险临床检查中，利用更多的法官和 OSCE 站，以加强标准制定的多种方法相结合的有效性。