Skip to main content

Advertisement

SpringerLink
Log in

Leveraging vision and kinematics data to improve realism of biomechanic soft tissue simulation for robotic surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Surgical simulations play an increasingly important role in surgeon education and developing algorithms that enable robots to perform surgical subtasks. To model anatomy, finite element method (FEM) simulations have been held as the gold standard for calculating accurate soft tissue deformation. Unfortunately, their accuracy is highly dependent on the simulation parameters, which can be difficult to obtain.

Methods

In this work, we investigate how live data acquired during any robotic endoscopic surgical procedure may be used to correct for inaccurate FEM simulation results. Since FEMs are calculated from initial parameters and cannot directly incorporate observations, we propose to add a correction factor that accounts for the discrepancy between simulation and observations. We train a network to predict this correction factor.

Results

To evaluate our method, we use an open-source da Vinci Surgical System to probe a soft tissue phantom and replay the interaction in simulation. We train the network to correct for the difference between the predicted mesh position and the measured point cloud. This results in 15–30% improvement in the mean distance, demonstrating the effectiveness of our approach across a large range of simulation parameters.

Conclusion

We show a first step towards a framework that synergistically combines the benefits of model-based simulation and real-time observations. It corrects discrepancies between simulation and the scene that results from inaccurate modeling parameters. This can provide a more accurate simulation environment for surgeons and better data with which to train algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Allard J, Cotin S, Faure F, Bensoussan P, Poyer F et al (2007) SOFA-an open source framework for medical simulation. Stud Health Technol Inform 125:13–18

    PubMed  Google Scholar 

  2. Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, vol. 1611, pp. 586–606. International Society for Optics and Photonics

  3. Bianchi G, Solenthaler B, Székely G, Harders M (2004) Simultaneous topology and stiffness identification for mass-spring models based on fem reference deformations. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 293–301

  4. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 424–432

  5. Finn C, Levine S (2017) Deep visual foresight for planning robot motion. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2786–2793

  6. Fontanelli GA, Selvaggio M, Ferro M, Ficuciello F, Vendittelli M, Siciliano B (2018) A v-rep simulator for the da vinci research kit robotic platform. In: 2018 7th IEEE international conference on biomedical robotics and biomechatronics (Biorob). IEEE, pp 1056–1061

  7. Geuzaine C, Remacle JF (2009) Gmsh: a 3-d finite element mesh generator with built-in pre-and post-processing facilities. Int J Numer Methods Eng 79(11):1309–1331

    Article  Google Scholar 

  8. Gondokaryono RA, Agrawal A, Munawar A, Nycz CJ, Fischer GS (2019) An approach to modeling closed-loop kinematic chain mechanisms, applied to simulations of the da vinci surgical system. Acta Polytech Hung 16(8):29–48

    Google Scholar 

  9. Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da vinci® surgical system. In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6434–6439

  10. Lee J, Lee S, Chang J, Thompson MS, Kang D, Park S, Park S (2013) A novel method for the accurate evaluation of poisson’s ratio of soft polymer materials. Sci World J 2013:43–52

    Google Scholar 

  11. Liu X, Sinha A, Unberath M, Ishii M, Hager GD, Taylor RH, Reiter A (2018) Self-supervised learning for dense depth estimation in monocular endoscopy. In: Stoyanov D, et al. (eds) OR 2.0 context-aware operating theaters, computer assisted robotic endoscopy, clinical image-based procedures, and skin image analysis. Springer, Cham, pp.128–138

    Google Scholar 

  12. Meister F, Passerini T, Mihalef V, Tuysuzoglu A, Maier A, Mansi T (2020) Deep learning acceleration of total lagrangian explicit dynamics for soft tissue mechanics. Comput Methods Appl Mech Eng 358:112628

    Article  Google Scholar 

  13. Mendizabal A, Márquez-Neila P, Cotin S (2020) Simulation of hyperelastic materials in real-time using deep learning. Med Image Anal 59:101569

    Article  PubMed  Google Scholar 

  14. Morooka K, Chen X, Kurazume R, Uchida S, Hara K, Iwashita Y, Hashizume M (2008) Real-time nonlinear fem with neural network for simulating soft organ model deformation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 742–749

  15. Munawar A, Wang Y, Gondokaryono R, Fischer G (2019) A real-time dynamic simulator and an associated front-end representation format for simulating complex robots and environments. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE

  16. Pfeiffer M, Riediger C, Weitz J, Speidel S (2019) Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks. Int J Comput Assist Radiol Surg 14(7):1147–1155

    Article  PubMed  Google Scholar 

  17. Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) Ros: an open-source robot operating system. In: ICRA workshop on open source software, vol 3, p 5. Kobe, Japan

  18. Richter F, Orosco RK, Yip MC (2019) Open-sourced reinforcement learning environments for surgical robotics. arXiv preprint arXiv:1903.02090

  19. Roewer-Despres F, Khan N, Stavness I (2018) Towards finite element simulation using deep learning. In: 15th international symposium on computer methods in biomechanics and biomedical engineering

  20. Rusu RB, Cousins S (2011) 3d is here: Point cloud library (pcl). In: 2011 IEEE international conference on robotics and automation, pp 1–4. IEEE

  21. Shin C, Ferguson PW, Pedram SA, Ma J, Dutson EP, Rosen J (2019) Autonomous tissue manipulation via surgical robot using learning based model predictive control. In: 2019 international conference on robotics and automation (ICRA). IEEE, pp 3875–3881

  22. Talbot H, Haouchine N, Peterlik I, Dequidt J, Duriez C et al (2015) Surgery training, planning and guidance using the SOFA framework. Eurographics. Zurich, Switzerland

  23. Zhang J, Zhong Y, Gu C (2017) Deformable models for surgical simulation: a survey. IEEE Rev Biomed Eng 11:143–164

    Article  PubMed  Google Scholar 

Download references

Funding

This work is supported by National Science Foundation NRI 1637789. The Titan V used for this research was donated by the Nvidia Corporation.

Author information

Authors and Affiliations

  1. Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA

    Jie Ying Wu, Peter Kazanzides & Mathias Unberath

Authors
  1. Jie Ying Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Kazanzides
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mathias Unberath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Ying Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal participants

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, J.Y., Kazanzides, P. & Unberath, M. Leveraging vision and kinematics data to improve realism of biomechanic soft tissue simulation for robotic surgery. Int J CARS 15, 811–818 (2020). https://doi.org/10.1007/s11548-020-02139-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02139-6

Keywords

Search

Navigation