Purpose: 

Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery. In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios.

Methods: 

We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We furthermore present three state-of-the-art RGB-based pose estimation baselines.

Results: 

We evaluate three baseline models on the proposed datasets. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation.

Conclusion: 

To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications.

中文翻译：

---

迈向无标记手术工具和姿势估计

目的： 

在计算机辅助手术的背景下，工具和手术活动的跟踪变得越来越重要。在这项工作中，我们提出了一个数据生成框架，数据集和基线方法，以促进在现实手术场景中在无标记手和仪器姿势估计方向上的进一步研究。

方法： 

我们开发了一个渲染管道，以创建廉价且逼真的合成数据以进行模型预训练。随后，我们提出了在实验设置中使用手和物体姿势真实性来捕获和标记真实数据的管道，以收集高质量的真实数据。我们还介绍了三个基于RGB的最新姿态估计基线。

结果： 

我们在建议的数据集上评估了三个基线模型。性能最佳的基线在合成数据上实现的平均工具3D顶点误差为16.7 mm，在真实数据上实现的平均3D顶点误差为13.8 mm，与基于RGB的手/对象姿势估计中的最新技术相当。

结论： 

据我们所知，我们提出了第一个综合和真实数据生成管道，以生成开放手术的手和物体姿势标签。我们为基于RGB的对象和基于RGB帧的对象/手部姿势估计提供了三种基线模型。我们逼真的合成数据生成管道可能有助于克服外科领域的数据瓶颈，并且可以轻松地转移到其他医疗应用中。