This paper reports a new type of augmented reality (AR) system that integrates a Microsoft HoloLens device with a three-dimensional (3D) point tracking module for medical training and telementored surgery. In this system, a stereo camera is used to track the 3D position of a scalpel and transfer its coordinates wirelessly to a HoloLens device. In the scenario of surgical training, a virtual surgical scene with pre-recorded surgical annotations is superimposed with the actual surgical scene so that the surgical trainee is able to operate following virtual instructions. In the scenario of telementored surgery, the virtual surgical scene is co-registered with the actual surgical scene so that the virtual scalpel remotely mentored by an experienced surgeon provides the AR guidance for the inexperienced on-site operator. The performance characteristics of the proposed AR telementoring system are verified by benchtop experiments. The clinical applicability of the proposed system in telementored skin grafting surgery and fasciotomy is validated in a New Zealand rabbit model. Our benchtop and in vivo experiments demonstrate the potential to improve surgical performance and reduce healthcare disparities in remote areas with limited resources.

本文报告了一种新型增强现实 (AR) 系统，该系统将 Microsoft HoloLens 设备与三维 (3D) 点跟踪模块集成在一起，用于医疗培训和远程控制手术。在该系统中，立体相机用于跟踪手术刀的 3D 位置并将其坐标无线传输到 HoloLens 设备。在手术训练场景中，将预先录制好的手术注释的虚拟手术场景与实际手术场景叠加，使受训者能够按照虚拟指令进行操作。在远程指导手术场景中，虚拟手术场景与实际手术场景协同配准，由经验丰富的外科医生远程指导的虚拟手术刀为缺乏经验的现场操作者提供AR指导。所提出的 AR 远程监控系统的性能特征通过台式实验进行了验证。所提出的系统在远程皮肤移植手术和筋膜切开术中的临床适用性在新西兰兔模型中得到验证。我们的台式和体内实验证明了在资源有限的偏远地区改善手术性能和减少医疗保健差异的潜力。