Abstract
Purpose
Accurate needle tracking provides essential information for MRI-guided percutaneous interventions. Passive needle tracking using MR images is challenged by variations of the needle-induced signal void feature in different situations. This work aimed to develop an automatic needle tracking algorithm for MRI-guided interventions based on the Mask Region Proposal-Based Convolutional Neural Network (R-CNN).
Methods
Mask R-CNN was adapted and trained to segment the needle feature using 250 intra-procedural images from 85 MRI-guided prostate biopsy cases and 180 real-time images from MRI-guided needle insertion in ex vivo tissue. The segmentation masks were passed into the needle feature localization algorithm to extract the needle feature tip location and axis orientation. The proposed algorithm was tested using 208 intra-procedural images from 40 MRI-guided prostate biopsy cases, and 3 real-time MRI datasets in ex vivo tissue. The algorithm results were compared with human-annotated references.
Results
In prostate datasets, the proposed algorithm achieved needle feature tip localization error with median Euclidean distance (dxy) of 0.71 mm and median difference in axis orientation angle (dθ) of 1.28°, respectively. In 3 real-time MRI datasets, the proposed algorithm achieved consistent dynamic needle feature tracking performance with processing time of 75 ms/image: (a) median dxy = 0.90 mm, median dθ = 1.53°; (b) median dxy = 1.31 mm, median dθ = 1.9°; (c) median dxy = 1.09 mm, median dθ = 0.91°.
Conclusions
The proposed algorithm using Mask R-CNN can accurately track the needle feature tip and axis on MR images from in vivo intra-procedural prostate biopsy cases and ex vivo real-time MRI experiments with a range of different conditions. The algorithm achieved pixel-level tracking accuracy in real time and has potential to assist MRI-guided percutaneous interventions.
Similar content being viewed by others
References
Gupta S, Madoff DC (2007) Image-guided percutaneous needle biopsy in cancer diagnosis and staging. Tech Vasc Interv Radiol 10:88–101. https://doi.org/10.1053/j.tvir.2007.09.005
McWilliams JP, Lee EW, Yamamoto S, Loh CT, Kee ST (2010) Image-guided tumor ablation: emerging technologies and future directions. Semin Interv Radiol 27:302–313. https://doi.org/10.1055/s-0030-1261789
Campbell-Washburn AE, Faranesh AZ, Lederman RJ, Hansen MS (2015) Magnetic resonance sequences and rapid acquisition for MR-guided interventions. Magn Reson Imaging Clin 23:669–679. https://doi.org/10.1016/j.mric.2015.05.006
Campbell-Washburn AE, Tavallaei MA, Pop M, Grant EK, Chubb H, Rhode K, Wright GA (2017) Real-time MRI guidance of cardiac interventions. J Magn Reson Imaging 46:935–950. https://doi.org/10.1002/jmri.25749
DiMaio SP, Kacher D, Ellis R, Fichtinger G, Hata N, Zientara G, Panych L, Kikinis R, Jolesz F (2006) Needle artifact localization in 3T MR images. Stud Health Technol Inform 119:120–125
DiMaio SP, Samset E, Fischer G, Iordachita I, Fichtinger G, Jolesz F, Tempany CM (2007) Dynamic MRI scan plane control for passive tracking of instruments and devices. In: Medical image computing and computer-assisted intervention (MICCAI), pp 50–58. https://doi.org/10.1007/978-3-540-75759-7_7
Görlitz RA, Tokuda J, Hoge SW, Chu R, Panych LP, Tempany C, Hata N (2010) Development and validation of a real-time reduced field of view imaging driven by automated needle detection for MRI-guided interventions. In: SPIE medical imaging, pp 762515–762519. https://doi.org/10.1117/12.840837
Zijlstra F, Bouwman JG, Braškutė I, Viergever MA, Seevinck PR (2016) Fast Fourier-based simulation of off-resonance artifacts in steady-state gradient echo MRI applied to metal object localization. Magn Reson Med 78:2035–2041. https://doi.org/10.1002/mrm.26556
Moore CM, Robertson NL, Arsanious N, Middleton T, Villers A, Klotz L, Taneja SS, Emberton M (2013) Image-guided prostate biopsy using magnetic resonance imaging–derived targets: a systematic review. Eur Urol 63:125–140. https://doi.org/10.1016/j.eururo.2012.06.004
Tan N, Lin W-C, Khoshnoodi P, Asvadi NH, Yoshida J, Margolis DJA, Lu DSK, Wu H, Sung KH, Lu DY, Huang J, Raman SS (2016) In-bore 3-T MR-guided transrectal targeted prostate biopsy: prostate imaging reporting and data system version 2–based diagnostic performance for detection of prostate cancer. Radiology 283:130–139. https://doi.org/10.1148/radiol.2016152827
Verma S, Choyke PL, Eberhardt SC, Oto A, Tempany CM, Turkbey B, Rosenkrantz AB (2017) The current state of MR imaging–targeted biopsy techniques for detection of prostate cancer. Radiology 285:343–356. https://doi.org/10.1148/radiol.2017161684
Ladd ME, Erhart P, Debatin JF, Romanowski BJ, Boesiger P, McKinnon GC (1996) Biopsy needle susceptibility artifacts. Magn Reson Med 36:646–651. https://doi.org/10.1002/mrm.1910360423
Cole GA, Harrington K, Su H, Camilo A, Pilitsis JG, Fischer GS (2014) Closed-loop actuated surgical system utilizing real-time in situ MRI guidance. In: International symposium on experimental robotics (ISER), pp 785–798. https://doi.org/10.1007/978-3-642-28572-1_54
Mikaiel S, Simonelli J, Li X, Lee Y, Lee YS, Sung K, Lu D, Tsao TC, Wu HH (2020) MRI-guided targeted needle placement during motion using hydrostatic actuators. Int J Med Robot Comput Assist Surg 16(2):e2041. https://doi.org/10.1002/rcs.2041
Mehrtash A, Ghafoorian M, Pernelle G, Ziaei A, Heslinga FG, Tuncali K, Fedorov A, Kikinis R, Tempany CM, Wells WM, Abolmaesumi P, Kapur T (2019) Automatic needle segmentation and localization in MRI with 3D convolutional neural networks: application to MRI-targeted prostate biopsy. IEEE Trans Med Imaging 38:1026–1036. https://doi.org/10.1109/TMI.2018.2876796
Zijlstra F, Viergever MA, Seevinck PR (2019) SMART tracking: simultaneous anatomical imaging and real-time passive device tracking for MR-guided interventions. Phys Med 64:252–260. https://doi.org/10.1016/j.ejmp.2019.07.019
Ho M, Kim Y, Cheng SS, Gullapalli R, Desai JP (2015) Design, development, and evaluation of an MRI-guided SMA spring-actuated neurosurgical robot. Int J Rob Res 34:1147–1163. https://doi.org/10.1177/0278364915579069
Bergeles C, Vartholomeos P, Qin L, Dupont PE (2013) Closed-loop commutation control of an MRI-powered robot actuator. In: 2013 IEEE international conference on robotics and automation, pp 698–703. https://doi.org/10.1109/ICRA.2013.6630649
Moreira P, Patel N, Wartenberg M, Li G, Tuncali K, Heffter T, Burdette EC, Iordachita I, Fischer GS, Hata N, Tempany CM, Tokuda J (2018) Evaluation of robot-assisted MRI-guided prostate biopsy: needle path analysis during clinical trials. Phys Med Biol 63:20NT02. https://doi.org/10.1088/1361-6560/aae214
Mwikirize C, Nosher JL, Hacihaliloglu I (2018) Signal attenuation maps for needle enhancement and localization in 2D ultrasound. Int J Comput Assist Radiol Surg 13:363–374. https://doi.org/10.1007/s11548-017-1698-y
Uherčík M, Kybic J, Liebgott H, Cachard C (2010) Model fitting using RANSAC for surgical tool localization in 3-D ultrasound images. IEEE Trans Biomed Eng 57:1907–1916. https://doi.org/10.1109/TBME.2010.2046416
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge
Pourtaherian A, Ghazvinian Zanjani F, Zinger S, Mihajlovic N, Ng G, Korsten H, de With P (2017) Improving needle detection in 3D ultrasound using orthogonal-plane convolutional networks. In: Medical image computing and computer-assisted intervention (MICCAI), pp 610–618. https://doi.org/10.1007/978-3-319-66185-8_69
Weine J, Breton E, Garnon J, Gangi A, Maier F (2019) Deep learning based needle localization on real-time MR images of patients acquired during MR-guided percutaneous interventions. In: Proceedings of the ISMRM 27th annual meeting, p 973
He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN. In: International conference on computer vision (ICCV), pp 2980–2988. https://doi.org/10.1109/ICCV.2017.322
Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: International conference on neural information processing systems (NIPS), pp 91–99. https://doi.org/10.1109/TPAMI.2016.2577031
Vuola AO, Akram SU, Kannala J (2019) Mask-RCNN and U-Net ensembled for nuclei segmentation. In: IEEE international conference on biomedical imaging (ISBI), pp 208–212. https://doi.org/10.1109/isbi.2019.8759574
Li X, Raman SS, Lu D, Lee Y, Tsao T, Wu HH (2019) Real-time needle detection and segmentation using Mask R-CNN for MRI-guided interventions. In: Proceedings of the ISMRM 27th annual meeting, p 972
Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. GitHub repository
Li X, Lee Y-H, Mikaiel S, Simonelli J, Tsao TC, Wu HH (2020) Respiratory motion prediction using fusion-based multi-rate Kalman filtering and real-time golden-angle radial MRI. IEEE Trans Biomed Eng 67(6):1727–1738. https://doi.org/10.1109/TBME.2019.2944803
Hansen MS, Sørensen TS (2013) Gadgetron: an open source framework for medical image reconstruction. Magn Reson Med 69:1768–1776. https://doi.org/10.1002/mrm.24389
Sørensen TS, Atkinson D, Schaeffter T, Hansen MS (2009) Real-time reconstruction of sensitivity encoded radial magnetic resonance imaging using a graphics processing unit. IEEE Trans Med Imaging 28:1974–1985. https://doi.org/10.1109/TMI.2009.2027118
Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) European conference on computer vision (ECCV). Springer, Cham, pp 740–755. https://doi.org/10.1007/978-3-319-10602-1_48
Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: International conference on neural information processing systems (NIPS), pp 3320–3328
Boggs PT, Rogers JE (1990) Orthogonal distance regression. Contemp Math 112:183–194. https://doi.org/10.6028/nist.ir.89-4197
Patel NA, van Katwijk T, Gang Li, Moreira P, Weijian Shang, Misra S, Fischer GS (2015) Closed-loop asymmetric-tip needle steering under continuous intraoperative MRI guidance. In: IEEE engineering in medicine and biology society (EMBC), pp 4869–4874. https://doi.org/10.1109/EMBC.2015.7319484
Kim YK, Kim YK, Park HJ, Park MJ, Lee WJ, Choi D (2014) Noncontrast MRI with diffusion-weighted imaging as the sole imaging modality for detecting liver malignancy in patients with high risk for hepatocellular carcinoma. Magn Reson Imaging 32:610–618. https://doi.org/10.1016/j.mri.2013.12.021
Stamey TA, Freiha FS, McNeal JE, Redwine EA, Whittemore AS, Schmid HP (1993) Localized prostate cancer. Relationship of tumor volume to clinical significance for treatment of prostate cancer. Cancer 71:933–938. https://doi.org/10.1002/1097-0142(19930201)71:3+%3c933:aid-cncr2820711408%3e3.0.co;2-l
Renfrew M, Griswold M, Çavusoglu MC (2018) Active localization and tracking of needle and target in robotic image-guided intervention systems. Auton Robots 42:83–97. https://doi.org/10.1007/s10514-017-9640-2
Song S-E, Cho NB, Iordachita II, Guion P, Fichtinger G, Whitcomb LL (2011) A study of needle image artifact localization in confirmation imaging of MRI-guided robotic prostate biopsy. In: IEEE international conference on robotics and automation (ICRA), pp 4834–4839. https://doi.org/10.1109/ICRA.2011.5980309
Acknowledgements
This research received funding support from Siemens Healthineers and the Department of Radiological Sciences at UCLA. The authors appreciate helpful discussions about the manuscript writing with the UCLA Graduate Writing Center and Dr. Le Zhang.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
This study was supported in part by Siemens Healthineers. Siemens Healthineers had no role in study design, data collection and analysis, or preparation of this manuscript.
Research involving human participants and informed consent
Prostate datasets were analyzed under an institutional review board approved and HIPAA-compliant retrospective study protocol with a waiver of consent.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Needle feature segmentation (red) and tip tracking (green) videos corresponding to the results reported in Table 4a. The videos are played back with the actual image acquisition rate of 15 frames/s (temporal resolution of 67 ms/frame) (MP4 22219 kb).
Needle feature segmentation (red) and tip (green) tracking videos corresponding to the results reported in Table 4b. The videos are played back with the actual image acquisition rate of 12 frames/s (temporal resolution of 84 ms/frame) (MP4 51805 kb).
Needle feature segmentation (red) and tip (green) tracking videos corresponding to the results reported in Table 4c. The videos are played back with the actual image acquisition rate of 12 frames/s (temporal resolution of 84 ms/frame) (MP4 35742 kb).
Rights and permissions
About this article
Cite this article
Li, X., Young, A.S., Raman, S.S. et al. Automatic needle tracking using Mask R-CNN for MRI-guided percutaneous interventions. Int J CARS 15, 1673–1684 (2020). https://doi.org/10.1007/s11548-020-02226-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-020-02226-8