Real-time tool tracking in minimally invasive-surgery (MIS) has numerous applications for computer-assisted interventions (CAIs). Visual tracking approaches are a promising solution to real-time surgical tool tracking, however, many approaches may fail to complete tracking when the tracker suffers from issues such as motion blur, adverse lighting, specular reflections, shadows, and occlusions. We propose an automatic real-time method for two-dimensional tool detection and tracking based on a spatial transformer network (STN) and spatio-temporal context (STC). Our method exploits both the ability of a convolutional neural network (CNN) with an in-house trained STN and STC to accurately locate the tool at high speed. Then we compared our method experimentally with other four general of CAIs’ visual tracking methods using eight existing online and in-house datasets, covering both in vivo abdominal, cardiac and retinal clinical cases in which different surgical instruments were employed. The experiments demonstrate that our method achieved great performance with respect to the accuracy and the speed. It can track a surgical tool without labels in real time in the most challenging of cases, with an accuracy that is equal to and sometimes surpasses most state-of-the-art tracking algorithms. Further improvements to our method will focus on conditions of occlusion and multi-instruments.

微创手术（MIS）中的实时工具跟踪在计算机辅助干预（CAI）中具有众多应用。视觉跟踪方法是实时手术工具跟踪的一种有前途的解决方案，但是，当跟踪器遇到诸如运动模糊，不利照明，镜面反射，阴影和遮挡等问题时，许多方法可能无法完成跟踪。我们提出了一种基于空间互感器网络（STN）和时空上下文（STC）的二维工具检测和跟踪的自动实时方法。我们的方法利用了经过内部训练的STN和STC的卷积神经网络（CNN）的能力来高速准确地定位工具。体内腹部，心脏和视网膜的临床案例，其中采用了不同的手术器械。实验表明，我们的方法在精度和速度上都取得了很好的性能。在最具挑战性的情况下，它可以实时跟踪没有标签的手术工具，其准确度有时甚至超过大多数最新的跟踪算法。我们方法的进一步改进将集中在闭塞和多器械的条件上。