Purpose

Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage.

Methods

A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12–15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation.

Results

Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery.

Conclusion

The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon’s skill variability in addition to their mean score to have proper knowledge of a surgeon’s overall skill level.

中文翻译：

---

实际机器人手术中手术技术技能知觉的时间变异性。

目的

来自非专家，教师外科医生或自动化绩效指标的摘要分数指标已被公认为是报告外科医生技术技能的主要方法。本文的目的是了解在长时间的外科手术录像中，外科医生的技术技能评估中是否存在重大波动。

方法

一组12个来自普通人类机器人手术的机器人手术病例的视频被用来评估在手术视频的每一分钟中感知的技术技能，这些视频原本的长度为12-15分钟。每个视频的线性混合效果模型用于将每分钟的评分与其他每分钟的评分进行比较，以了解除评分者间和评分者间的差异之外，是否可以检测并可靠地测量分数随时间的变化。

结果

对机器人技能得分的全球评估评估随时间的变化进行建模，为12位外科医生中的11位的预测模型做出了重要贡献。这表明在机器人手术期间，随着时间的流逝，技术技能发生了可测量的变化。

结论

这项研究的发现提出了有关需要评估的最佳镜头时长的问题，以便对更长的手术时间准确评估手术技术技能。对于有监督的机器学习方法，这可能意味着不可忽略的标签噪声。将来，可能需要报告外科医生的技能变异性（除了他们的平均得分以外），以适当了解外科医生的整体技能水平。