Skip to main content
SpringerLink
Log in

Transferring optimal contact skills to flexible manipulators by reinforcement learning

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

Flexible/soft manipulators have the potential to maneuver in confined space and reach deeply-seated targets via curvy trajectories, thus enjoy increasing popularity in minimally invasive surgery (MIS) community. We aim to automate palpation movement for this type of robots, an important procedure for disease diagnosis, where multiple force and pose requirements are to be achieved simultaneously. It’s challenging to obtain accurate models due to the system’s inherent nonlinearities and actuation hysteresis. Moreover, unknown contact transitions and high-dimensionality specific to the palpation task, pose great challenges to deriving optimal task policies. We employ the model-free reinforcement learning method for learning palpation skills through deterministic policy gradient, whose reward function was carefully shaped to accommodate all the task objectives. In addition, we design a safety check routine to avoid undesirable collisions and a dedicated initialization process for generalization to various environment conditions. We demonstrate successful implementation of the learning framework in simulation and real world. The trained policy succeeds in automating the designed tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://github.com/sparisi/mips.

References

  • Abidi, H., Gerboni, G., Brancadoro, M., Fras, J., Diodato, A., Cianchetti, M., Wurdemann, H., Althoefer, K., Menciassi, A.: Highly dexterous 2-module soft robot for intra-organ navigation in minimally invasive surgery. Int. J. Med. Robot. Comput. Assist. Surg. 14(1), e1875 (2018)

    Article  Google Scholar 

  • Abushagur, A.A., Arsad, N., Reaz, M.I., Bakar, A.: Advances in bio-tactile sensors for minimally invasive surgery using the fibre bragg grating force sensor technique: A survey. Sensors 14(4), 6633–6665 (2014)

    Article  Google Scholar 

  • Ahn, B., Park, K., Lee, H., Lorenzo, E.I.S., Rha, K.H., Kim, J.: Robotic palpation system for prostate cancer detection. In: Proceedings of the 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 644–649. IEEE (2010)

  • Ahn, B., Kim, Y., Oh, C.K., Kim, J.: Robotic palpation and mechanical property characterization for abnormal tissue localization. Med. Biol. Eng. Comput. 50(9), 961–971 (2012)

    Article  Google Scholar 

  • Ansari, Y., Manti, M., Falotico, E., Cianchetti, M., Laschi, C.: Multiobjective optimization for stiffness and position control in a soft robot arm module. IEEE Robot. Autom. Lett. 3(1), 108–115 (2018)

    Article  Google Scholar 

  • Burgner, J., Rucker, D.C., Gilbert, H.B., Swaney, P.J., Russell, P.T., Weaver, K.D., Webster, R.J.: A telerobotic system for transnasal surgery. IEEE/ASME Trans. Mechatron. 19(3), 996–1006 (2014)

    Article  Google Scholar 

  • Calinon, S., Bruno, D., Malekzadeh, M.S., Nanayakkara, T., Caldwell, D.G.: Human-robot skills transfer interfaces for a flexible surgical robot. Comput. Methods Progr. Biomed. 116(2), 81–96 (2014)

    Article  Google Scholar 

  • Chen, Y., Xu, W., Li, Z., Song, S., Lim, C.M., Wang, Y., Ren, H.: Safety-enhanced motion planning for flexible surgical manipulator using neural dynamics. IEEE Trans. Control Syst. Technol. PP(99), 1–13 (2016)

    Google Scholar 

  • Chen, F., Xu, W., Zhang, H., Wang, Y., Cao, J., Wang, M.Y., Ren, H., Zhu, J., Zhang, Y.: Topology optimized design, fabrication, and characterization of a soft cable-driven gripper. IEEE Robot. Autom. Lett. 3(3), 2463–2470 (2018)

    Article  Google Scholar 

  • Critch, A.: Toward negotiable reinforcement learning: shifting priorities in pareto optimal sequential decision-making (2017). arXiv:1701.01302 (arXiv preprint)

  • Garcıa, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 16(1), 1437–1480 (2015)

    MathSciNet  MATH  Google Scholar 

  • García, J., Iglesias, R., Rodríguez, M.A., Regueiro, C.V.: Incremental reinforcement learning for multi-objective robotic tasks. Knowl. Inf. Syst. 51(3), 911–940 (2017)

    Article  Google Scholar 

  • George Thuruthel, T., Falotico, E., Manti, M., Pratesi, A., Cianchetti, M., Laschi, C.: Learning closed loop kinematic controllers for continuum manipulators in unstructured environments. Soft Robot. 4(3), 285–296 (2017)

    Article  Google Scholar 

  • Gu, S., Holly, E., Lillicrap, T., Levine, S.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3389–3396. IEEE (2017)

  • Gupta, A., Eppner, C., Levine, S., Abbeel, P.: Learning dexterous manipulation for a soft robotic hand from human demonstrations. In: Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3786–3793. IEEE (2016)

  • Herzig, N., Maiolino, P., Iida, F., Nanayakkara, T.: A variable stiffness robotic probe for soft tissue palpation. IEEE Robot. Autom. Lett. 3(2), 1168–1175 (2018)

    Article  Google Scholar 

  • Hwangbo, J., Sa, I., Siegwart, R., Hutter, M.: Control of a quadrotor with reinforcement learning. IEEE Robot. Autom. Lett. 2(4), 2096–2103 (2017)

    Article  Google Scholar 

  • Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., Hutter, M.: Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4(26), eaau5872 (2019)

    Article  Google Scholar 

  • Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: A survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)

    Article  Google Scholar 

  • Konstantinova, J., Cotugno, G., Dasgupta, P., Althoefer, K., Nanayakkara, T.: Autonomous robotic palpation of soft tissue using the modulation of applied force. In: Proceedings of the 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 323–328. IEEE (2016)

  • Konstantinova, J., Jiang, A., Althoefer, K., Dasgupta, P., Nanayakkara, T.: Implementation of tactile sensing for palpation in robot-assisted minimally invasive surgery: a review. IEEE Sens. J. 14(8), 2490–2501 (2014)

    Article  Google Scholar 

  • Konstantinova, J., Cotugno, G., Dasgupta, P., Althoefer, K., Nanayakkara, T.: Palpation force modulation strategies to identify hard regions in soft tissue organs. PLoS One 12(2), e0171706 (2017)

    Article  Google Scholar 

  • Kwon, Y.S., Tae, K., Yi, B.J.: Suspension laryngoscopy using a curved-frame trans-oral robotic system. Int. J. Comput. Assist. Radiol. Surg. 9(4), 535–40 (2014)

    Article  Google Scholar 

  • Lee, K.H., Fu, D.K., Leong, M.C., Chow, M., Fu, H.C., Althoefer, K., Sze, K.Y., Yeung, C.K., Kwok, K.W.: Nonparametric online learning control for soft continuum robot: An enabling technique for effective endoscopic navigation. Soft Robot. 4(4), 324–337 (2017)

    Article  Google Scholar 

  • Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning (2015). arXiv:1509.02971 (arXiv preprint)

  • Ma, X., Wang, P., Ye, M., Chiu, P.W.Y., Li, Z.: Shared autonomy of a flexible manipulator in constrained endoluminal surgical tasks. IEEE Robot. Autom. Lett. 4(3), 3106–3112 (2019). https://doi.org/10.1109/LRA.2019.2924851

    Article  Google Scholar 

  • Malekzadeh, M.S., Bruno, D., Calinon, S., Nanayakkara, T., Caldwell, D.G.: Skills transfer across dissimilar robots by learning context-dependent rewards. In: Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1746–1751. IEEE (2013)

  • Malekzadeh, M.S., Calinon, S., Bruno, D., Caldwell, D.G.: Learning by imitation with the stiff-flop surgical robot: a biomimetic approach inspired by octopus movements. Robot. Biomim. 1(1), 13 (2014)

    Article  Google Scholar 

  • Nichols, K.A., Okamura, A.M.: Autonomous robotic palpation: Machine learning techniques to identify hard inclusions in soft tissues. In: Proceedings of the 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 4384–4389. IEEE (2013)

  • Nichols, K.A., Okamura, A.M.: Methods to segment hard inclusions in soft tissue during autonomous robotic palpation. IEEE Trans. Robot. 31(2), 344–354 (2015)

    Article  Google Scholar 

  • Osa, T., Sugita, N., Mitsuishi, M.: Online trajectory planning and force control for automation of surgical tasks. IEEE Trans. Autom. Sci. Eng. 15(2), 675–691 (2018)

    Article  Google Scholar 

  • Ottermo, M.V., Stavdahl, O., Johansen, T.A.: Palpation instrument for augmented minimally invasive surgery. In: Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), vol. 4, pp. 3960–3964. IEEE (2004)

  • Pham, T.H., De Magistris, G., Tachibana, R.: Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In: Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6236–6243. IEEE (2018)

  • Reichert, B., Stelzenmueller, W.: Palpation Techniques: Surface Anatomy for Physical Therapists. Thieme, Stuttgart (2011)

    Google Scholar 

  • Roy, N., Newman, P., Srinivasa, S.: Tendon-Driven Variable Impedance Control Using Reinforcement Learning. MITP (2013). https://ieeexplore.ieee.org/document/6577948

  • Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 387–395. PMLR, Bejing, China (2014)

  • Solodova, R.F., Galatenko, V.V., Nakashidze, E.R., Shapovalyants, S.G., Andreytsev, I.L., Sokolov, M.E., Podolskii, V.E.: Instrumental mechanoreceptoric palpation in gastrointestinal surgery. Minim. Invasive Surg. 2017, 6481856 (2017). https://doi.org/10.1155/2017/6481856

    Google Scholar 

  • Song, S., Li, Z., Yu, H., Ren, H.: Electromagnetic positioning for tip tracking and shape sensing of flexible robots. IEEE Sens. J. 15(8), 4565–4575 (2015)

    Article  Google Scholar 

  • Thananjeyan, B., Garg, A., Krishnan, S., Chen, C., Miller, L., Goldberg, K.: Multilateral surgical pattern cutting in 2d orthotropic gauze with deep reinforcement learning policies for tensioning. In: Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2371–2378. IEEE (2017)

  • Xu, W., Chen, J., Lau, H.Y., Ren, H.: Automate surgical tasks for a flexible serpentine manipulator via learning actuation space trajectory from demonstration. In: Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 4406–4413. IEEE (2016)

  • Xu, W., Chen, J., Lau, H.Y., Ren, H.: Data-driven methods towards learning the highly nonlinear inverse kinematics of tendon-driven surgical manipulators. Int. J. Med. Robot. Comput. Assist. Surg. 13(3), e1774 (2017)

    Article  Google Scholar 

  • Yip, M.C., Camarillo, D.B.: Model-less feedback control of continuum manipulators in constrained environments. IEEE Trans. Robot. 30(4), 880–889 (2014)

    Article  Google Scholar 

  • Yip, M.C., Camarillo, D.B.: Model-less hybrid position/force control: A minimalist approach for continuum manipulators in unknown, constrained environments. IEEE Robot. Autom. Lett. 1(2), 844–851 (2016)

    Article  Google Scholar 

  • You, X., Zhang, Y., Chen, X., Liu, X., Wang, Z., Jiang, H., Chen, X.: Model-free control for soft manipulators based on reinforcement learning. In: Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2909–2915. IEEE (2017)

  • Zhao, J., Zheng, X., Zheng, M., Shih, A.J., Xu, K.: An endoscopic continuum testbed for finalizing system characteristics of a surgical robot for notes procedures. In: Proceedings of the 2013 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 63–70. IEEE (2013)

Download references

Acknowledgements

This work is supported by the Singapore Academic Research Fund under Grant R-397-000-297-114. The authors would like to thank Feifei Chen for fabricating the soft finger.

Author information

Authors and Affiliations

  1. The Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore

    Wenjun Xu, Anqi Pan & Hongliang Ren

  2. The Department of Control Science & Engineering, Tongji University, Shanghai, China

    Anqi Pan

Authors
  1. Wenjun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anqi Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongliang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongliang Ren.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, W., Pan, A. & Ren, H. Transferring optimal contact skills to flexible manipulators by reinforcement learning. Int J Intell Robot Appl 3, 326–337 (2019). https://doi.org/10.1007/s41315-019-00101-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-019-00101-7

Keywords

Search

Navigation