Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education,British Journal of Anaesthesia

当前位置： X-MOL 学术 › Br. J. Anaesth. › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education
British Journal of Anaesthesia ( IF 9.1 ) Pub Date : 2022-01-10 , DOI: 10.1016/j.bja.2021.12.012
Alayne Kealey ₁ , Fahad Alam ₂ , Lisa A Bahrey ₃ , Clyde T Matava ₄ , Graham A McCreath ₅ , Catharine M Walsh ₆

Affiliation

Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada; Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada; Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; The Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; Department of Anesthesia, University Health Network-Toronto General Hospital, Toronto, ON, Canada.
Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada; Department of Anesthesia and Pain Medicine, Hospital for Sick Children, Toronto, ON, Canada.
SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada.
The Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Research Institute, Hospital for Sick Children, Toronto, ON, Canada; Division of Gastroenterology, Hepatology and Nutrition, Hospital for Sick Children, University of Toronto, Toronto, ON, Canada; Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; SickKids Learning Institute, Hospital for Sick Children, Toronto, ON, Canada.

Background

Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

Methods

The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

Results

We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test–retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

Conclusions

This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

中文翻译：

麻醉临床遭遇评估 (ACEA) 工具支持基于能力的医学教育的有效性证据

背景

基于工作场所的评估 (WBA) 是基于能力的评估策略的关键。随着我们计划推出基于能力的医学教育，我们开发了一种基于委托的 WBA，即麻醉临床遭遇评估 (ACEA)，以评估独立实践对围手术期患者护理至关重要的能力的准备情况。本研究旨在检验 ACEA 在研究生麻醉学培训期间的有效性证据。

方法

ACEA 包括八项全球评级量表 (GRS)、整体独立性评级、八项清单和案例详细信息。从 2017 年 7 月至 2020 年 1 月，从该计划的在线评估门户网站中提取了多伦多大学麻醉住院医师的 ACEA 数据。效度证据是根据 Messick 的效度框架生成的，包括响应过程、内部结构、与其他变量的关系以及后果。

结果

我们分析了由 342 名评估员完成的 137 名居民的 8664 次评估。从普遍性分析来看，10 次独立观察（五名评估员各进行两次评估）足以实现训练中评估的 ≥0.70 的可靠性阈值。3.65/5 的综合 GRS 评分为确定接受操作者特征曲线分析的委托提供了最佳灵敏度 (93.6%) 和特异性 (90.8%)。重测信度高（组内相关系数 [ICC _2,1 ]=0.81）对于 14 天内的匹配评估。综合 GRS 分数在不同训练水平的居民之间存在显着差异（P <0.0001），并且与整体独立性高度相关（0.91，P<0.001)。GRS的内部一致性（α=0.96）非常好。