As surgical robots become more common, automating away some of the burden of complex direct human operation becomes ever more feasible. Model-free reinforcement learning (RL) is a promising direction toward generalizable automated surgical performance, but progress has been slowed by the lack of efficient and realistic learning environments. In this paper, we describe adding reinforcement learning support to the da Vinci Skill Simulator, a training simulation used around the world to allow surgeons to learn and rehearse technical skills. We successfully teach an RL-based agent to perform sub-tasks in the simulator environment, using either image or state data. As far as we know, this is the first time an RL-based agent is taught from visual data in a surgical robotics environment. Additionally, we tackle the sample inefficiency of RL using a simple-to-implement system which we term hybrid-batch learning (HBL), effectively adding a second, long-term replay buffer to the Q-learning process. Additionally, this allows us to bootstrap learning from images from the data collected using the easier task of learning from state. We show that HBL decreases our learning times significantly.

中文翻译：

---

具有精益强化学习的机器人手术

随着外科手术机器人变得越来越普遍，使复杂的直接人类操作的一些负担自动化可以变得越来越可行。无模型强化学习（RL）是朝着通用的自动化外科手术性能发展的有希望的方向，但是由于缺乏有效和现实的学习环境，进度已经放缓。在本文中，我们描述了在达芬奇技能模拟器中增加强化学习支持，达芬奇技能模拟器是一种在世界范围内使用的培训模拟，它允许外科医生学习和练习技术技能。我们成功地教导了一个基于RL的代理，以使用图像或状态数据在模拟器环境中执行子任务。据我们所知，这是第一次在外科手术机器人环境中从视觉数据中教授基于RL的代理。此外，我们使用简单易用的系统（称为混合批处理学习（HBL））解决了RL样本效率低下的问题，有效地为Q学习过程添加了第二个长期重播缓冲区。此外，这使我们可以使用从状态学习的简单任务，从收集的数据中的图像中引导学习。我们证明，HBL大大减少了我们的学习时间。