Skip to main content

Advertisement

SpringerLink
Log in

Deep learning-based liver segmentation for fusion-guided intervention

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Tumors often have different imaging properties, and there is no single imaging modality that can visualize all tumors. In CT-guided needle placement procedures, image fusion (e.g. with MRI, PET, or contrast CT) is often used as image guidance when the tumor is not directly visible in CT. In order to achieve image fusion, interventional CT image needs to be registered to an imaging modality, in which the tumor is visible. However, multi-modality image registration is a very challenging problem. In this work, we develop a deep learning-based liver segmentation algorithm and use the segmented surfaces to assist image fusion with the applications in guided needle placement procedures for diagnosing and treating liver tumors.

Methods

The developed segmentation method integrates multi-scale input and multi-scale output features in one single network for context information abstraction. The automatic segmentation results are used to register an interventional CT with a diagnostic image. The registration helps visualize the target and guide the interventional operation.

Results

The segmentation results demonstrated that the developed segmentation method is highly accurate with Dice of 96.1% on 70 CT scans provided by LiTS challenge. The segmentation algorithm is then applied to a set of images acquired for liver tumor intervention for surface-based image fusion. The effectiveness of the proposed methods is demonstrated through a number of clinical cases.

Conclusion

Our study shows that deep learning-based image segmentation can obtain useful results to help image fusion for interventional guidance. Such a technique may lead to a number of other potential applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://competitions.codalab.org/competitions/15595.

  2. https://competitions.codalab.org/competitions/17094

References

  1. Abi-Jaoudeh N, Kruecker J, Kadoury S, Kobeiter H, Venkatesan AM, Levy E, Wood BJ (2012) Multimodality image fusion-guided procedures: technique, accuracy, and applications. Cardio Vasc Interv Radiol 35(5):986–998. https://doi.org/10.1007/s00270-012-0446-5

    Article  Google Scholar 

  2. Besl PJ, McKay ND (1992) A method for registration of 3-d shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256. https://doi.org/10.1109/34.121791

    Article  Google Scholar 

  3. Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, Fu CW, Han X, Heng PA, Hesser J, Kadoury S, Konopczynski T, Le M, Li C, Li X, Lipkovà J, Lowengrub J, Meine H, Moltz JH, Pal C, Piraud M, Qi X, Qi J, Rempfler M, Roth K, Schenk A, Sekuboyina A, Vorontsov E, Zhou P, Hülsemeyer C, Beetz M, Ettlinger F, Gruen F, Kaissis G, Lohöfer F, Braren R, Holch J, Hofmann F, Sommer W, Heinemann V, Jacobs C, Mamani GEH, van Ginneken B, Chartrand G, Tang A, Drozdzal M, BenCohen A, Klang E, Amitai MM, Konen E, Greenspan H, Moreau J, Hostettler A, Soler L, Vivanti R, Szeskin A, Lev-Cohain N, Sosna J, Joskowicz L, Menze BH (2019) The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056

  4. Billings SD, Boctor EM, Taylor RH (2015) Iterative most-likely point registration (IMLP): a robust algorithm for computing optimal shape alignment. PloS One 10(3):e0117688

    Article  Google Scholar 

  5. Chen X, Zhang R, Yan P (2019) Feature fusion encoder decoder network for automatic liver lesion segmentation. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), pp 430–433, https://doi.org/10.1109/ISBI.2019.8759555

  6. Fang X, Du B, Xu S, Wood BJ, Yan P (2019) Unified multi-scale feature abstraction for medical image segmentation. arXiv preprint arXiv:1910.11456

  7. Haaga JR (2005) Interventional ct: 30 years’ experience. Eur Radiol Suppl 15(4):d116–d120

    Article  Google Scholar 

  8. Han X (2017) Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv:1704.07239

  9. Haskins G, Kruger U, Yan P (2019) Deep learning in medical image registration: a survey. arXiv preprint arXiv:1903.02026

  10. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  12. Herring JL, Dawant BM, Maurer CR, Muratore DM, Galloway RL, Fitzpatrick JM (1998) Surface-based registration of ct images to physical space for image-guided surgery of the spine: a sensitivity study. IEEE Transactions on Medical Imaging 17(5):743–752. https://doi.org/10.1109/42.736029

    Article  CAS  PubMed  Google Scholar 

  13. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: CVPR, pp 2261–2269

  14. Jones AK, Dixon RG, Collins JD, Walser EM, Nikolic B (2018) Best practice guidelines for ct-guided interventional procedures. J Vasc Interv Radiol 29(4):518

    Article  Google Scholar 

  15. Kikinis R, Pieper SD, Vosburgh KG (2014) 3d slicer: a platform for subject-specific image analysis, visualization, and clinical support. In: Intraoperative imaging and image-guided therapy. Springer, New York, NY, pp 277–289

  16. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25, Curran Associates, Inc., pp 1097–1105, http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

  17. Li X, Chen H, Qi X, Dou Q, Fu C, Heng P (2018) H-denseunet: hybrid densely connected unet for liver and tumor segmentation from CT volumes. IEEE Trans Med Imag 37(12):2663–2674. https://doi.org/10.1109/TMI.2018.2845918

    Article  Google Scholar 

  18. Liu S, Xu D, Zhou SK, Pauly O, Grbic S, Mertelmeier T, Wicklein J, Jerebko A, Cai W, Comaniciu D (2018) 3d anisotropic hybrid network: transferring convolutional features from 2d images to 3d anisotropic volumes. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 851–858

  19. Maintz J, Viergever MA (1998) A survey of medical image registration. Med Image Anal 2(1):1–36. https://doi.org/10.1016/S1361-8415(01)80026-8

    Article  CAS  PubMed  Google Scholar 

  20. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. In: NIPS 2017 Workshop Autodiff

  21. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, pp 234–241

  22. Vorontsov E, Tang A, Pal C, Kadoury S (2018) Liver lesion segmentation informed by joint liver segmentation. In: ISBI, pp 1332–1335, https://doi.org/10.1109/ISBI.2018.8363817

  23. Wang L, Lee CY, Tu Z, Lazebnik S (2015) Training deeper convolutional networks with deep supervision. arXiv preprint arXiv:1505.02496

  24. Yuan Y (2017) Hierarchical convolutional-deconvolutional neural networks for automatic liver and tumor segmentation. arXiv:1710.04540

Download references

Acknowledgements

We would also like to thank NVIDIA Corporation for the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering and Center for Biotechnology and Interdisciplinary Studies, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA

    Xi Fang & Pingkun Yan

  2. Center for Interventional Oncology, Radiology and Imaging Sciences, National Institutes of Health, Bethesda, MD, 20892, USA

    Sheng Xu & Bradford J. Wood

Authors
  1. Xi Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sheng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bradford J. Wood
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pingkun Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pingkun Yan.

Ethics declarations

Conflict of interest

NIH and RPI share intellectual property in the field, and one author receives royalties for licensed patents (BW).

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee of where the studies were conducted.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health (NIH) under Awards R21EB028001 and R01EB027898, and through an NIH Bench-to-Bedside award made possible by the National Cancer Institute.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, X., Xu, S., Wood, B.J. et al. Deep learning-based liver segmentation for fusion-guided intervention. Int J CARS 15, 963–972 (2020). https://doi.org/10.1007/s11548-020-02147-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02147-6

Keywords

Search

Navigation