The generous application of robot-assisted minimally invasive surgery (RAMIS) promotes human-machine interaction (HMI). Identifying various behaviors of doctors can enhance the RAMIS procedure for the redundant robot. It bridges intelligent robot control and activity recognition strategies in the operating room, including hand gestures and human activities. In this paper, to enhance identification in a dynamic situation, we propose a multimodal data fusion framework to provide multiple information for accuracy enhancement. Firstly, a multi-sensors based hardware structure is designed to capture varied data from various devices, including depth camera and smartphone. Furthermore, in different surgical tasks, the robot control mechanism can shift automatically. The experimental results evaluate the efficiency of developing the multimodal framework for RAMIS by comparing it with a single sensor system. Implementing the KUKA LWR4+ in a surgical robot environment indicates that the surgical robot systems can work with medical staff in the future.

机器人辅助微创手术（RAMIS）的大量应用促进了人机交互（HMI）。识别医生的各种行为可以增强冗余机器人的RAMIS程序。它在手术室中桥接了智能机器人控制和活动识别策略，包括手势和人类活动。在本文中，为了增强动态情况下的识别能力，我们提出了一种多模式数据融合框架，该框架可提供多个信息以提高准确性。首先，基于多传感器的硬件结构被设计为捕获来自各种设备（包括深度相机和智能手机）的各种数据。此外，在不同的手术任务中，机器人控制机构可以自动移动。实验结果通过与单传感器系统进行比较，评估了开发RAMIS多模式框架的效率。在外科手术机器人环境中实施KUKA LWR4 +表示外科手术机器人系统将来可以与医务人员一起使用。