Objective: 

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

Summary of Background Data: 

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

Methods: 

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

Results: 

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 – 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5–15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

Conclusions: 

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

中文翻译：

---

普通外科手术特定能力评估工具的有效性证据：范围界定审查

客观的： 

本研究旨在对支持普通外科手术特定手术评估工具的有效性证据提供重点和详细的评估。

背景数据摘要： 

基于能力的评估工具应得到强有力的有效性证据的支持，以便可靠地用于评估操作技能。当代有效性框架依赖于五个证据来源：内容、反应过程、内部结构、与其他变量的关系以及后果。

方法： 

对 8 个数据库进行了系统检索，以查找包含普通外科手术特定手术评估工具的研究。支持每个工具的有效性证据均根据当代有效性框架进行评估和评分。研究方法的严谨性通过医学教育研究质量工具进行评估。每个工具的教育效用均通过研究生医学教育认证委员会框架进行评估。

结果： 

有 28 项研究符合纳入标准，并评估了 23 种独特的工具。不同工具的有效性证据分数差异很大，范围为 3 – 14（最多 15）。评估研究方法质量的医学教育研究质量工具分数也各不相同（8.5–15.5，最高 16.5）。教育效用标准的直接报告是有限的。

结论： 

这项研究确定了一小部分普通外科手术特定的手术评估工具。其中许多工具的有效性证据有限，并且尚未经过充分研究，无法可靠地用于高风险的总结性评估。随着普通外科过渡到基于能力的培训，将需要更强大的手术评估工具库来支持住院医师教育和评估。