This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.

中文翻译：

---

daVinciNet：机器人辅助手术中运动和手术状态的联合预测

本文提出了一种使用多个输入源同时和联合预测手术器械的未来轨迹和机器人辅助手术 (RAS) 中手术子任务的未来状态的技术。这种预测是实现手术子任务的共享控制和监督自主性的必要的第一步。分钟级的手术子任务，例如缝合或超声扫描，通常具有可区分的工具运动学和视觉特征，并且可以描述为具有过渡示意图的一系列细粒度状态。我们提出了 daVinciNet——一种用于机器人运动和手术状态预测的端到端双任务模型。daVinciNet 使用从多个数据流（包括机器人运动学、内窥镜视觉、和系统事件。我们在 da Vinci Xi 手术系统和 JHU-ISI 手势和技能评估工作集 (JIGSAWS) 上收集的扩展机器人术中超声 (RIOUS+) 成像数据集上评估我们提出的模型。我们的模型实现了高达 93.85% 的短期（0.5s）和 82.11% 的长期（2s）状态预测精度，以及 1.07mm 的短期和 5.62mm 的长期轨迹预测误差。