Skip to main content
SpringerLink
Log in

Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Visualization of the cochlea is impossible due to the delicate and intricate ear anatomy. Augmented reality may be used to perform auditory nerve implantation by transmodiolar approach in patients with profound hearing loss.

Methods

We present an augmented reality system for the visualization of the cochlear axis in surgical videos. The system starts with an automatic anatomical landmark detection in preoperative computed tomography images based on deep reinforcement learning. These landmarks are used to register the preoperative geometry with the real-time microscopic video captured inside the auditory canal. Three-dimensional pose of the cochlear axis is determined using the registration projection matrices. In addition, the patient microscope movements are tracked using an image feature-based tracking process.

Results

The landmark detection stage yielded an average localization error of \(2.18 \pm 1.44\) mm (\(n = 8\)). The target registration error was \(0.31 \pm 0.10\) mm for the cochlear apex and \(15.10 \pm 1.28 ^{\circ }\) for the cochlear axis.

Conclusion

We developed an augmented reality system to visualize the cochlear axis in intraoperative videos. The system yielded millimetric accuracy and remained stable throughout the experimental study despite camera movements throughout the procedure in experimental conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. National Institute of Health (2019). https://report.nih.gov/nihfactsheets/viewfactsheet.aspx?csid=83. Accessed 28 Oct 2019

  2. Doyle JH, Doyle JB Jr, Turnbull FM Jr (1964) Electrical stimulation of eighth cranial nerve. Arch Otolaryngol 80:388–391. https://doi.org/10.1001/archotol.1964.00750040400005

    Article  CAS  PubMed  Google Scholar 

  3. Middlebrooks JC, Snyder RL (2007) Auditory prosthesis with a penetrating nerve array. J Assoc Res Otolaryngol 8:258–279. https://doi.org/10.1007/s10162-007-0070-2

    Article  PubMed  PubMed Central  Google Scholar 

  4. Guigou C, Leterme G, Pasquis B, Martin L, Tourrel G, Bretillon L, Grayeli AB (2016) Implantation of the auditory nerve via the middle ear cavity in rats with partial hearing preservation. Audiol Neurootol 21(2):98–104. https://doi.org/10.1159/000444005

    Article  CAS  PubMed  Google Scholar 

  5. Simmons FB, Epley JM, Lummis RC, Guttman N, Frishkopf LS, Harmon LD, Zwicker E (1965) Auditory nerve: electrical stimulation in man. Science 148(3666):104–106. https://doi.org/10.1126/science.148.3666.104

    Article  CAS  PubMed  Google Scholar 

  6. Afifi WFS, Guigou C, Mazalaigue S, Camuset JP, Ricolfi F, Grayeli AB (2015) Navigation-guided transmodiolar approach for auditory nerve implantation via the middle ear in humans. Audiol Neurootol 20(2):128–135. https://doi.org/10.1159/000366513

    Article  Google Scholar 

  7. Hussain R, Lalande A, Guigou C, Grayeli AB (2019) Contribution of augmented reality to minimally invasive computer-assisted cranial base surgery. IEEE J Biomed Health Inform (Early Access). https://doi.org/10.1109/JBHI.2019.2954003

    Article  Google Scholar 

  8. Hussain R, Lalande A, Marroquin R, Girum KB, Guigou C, Bozorg-Grayeli A (2018) Real-time augmented reality for ear surgery. In: Frangi A, Schnabel J, Davatzikos C, Alberola-López C, Fichtinger G (eds) Medical image computing and computer assisted intervention—MICCAI 2018, vol 11073. Lecture notes in computer science. Springer, Berlin, pp 324–331. https://doi.org/10.1007/978-3-030-00937-3_38

    Chapter  Google Scholar 

  9. Liu WP, Azizian M, Sorger J, Taylor RH, Reilly BK, Cleary K, Preciado D (2014) Cadaveric feasibility study of da vinci si-assisted cochlear implant with augmented visual navigation for otologic surgery. JAMA Otolaryngol Head Neck Surg 140(3):208–214. https://doi.org/10.1001/jamaoto.2013.6443

    Article  PubMed  Google Scholar 

  10. Cho NH, Jang JH, Jung W, Kim J (2014) In vivo imaging of middle-ear and inner-ear microstructures of a mouse guided by SD-OCT combined with a surgical microscope. Opt Express 22(8):8985–8995. https://doi.org/10.1364/OE.22.008985

    Article  PubMed  PubMed Central  Google Scholar 

  11. Wong K, Yee HM, Xavier BA, Grillone GA (2018) Applications of augmented reality in otolaryngology: a systematic review. Otolaryngol Head Neck Surg 159(6):956–967. https://doi.org/10.1177/0194599818796476

    Article  PubMed  Google Scholar 

  12. Alansary A, Oktay O, Li Y, Le Folgoc L, Hou B, Vaillant G, Kamnitsas K, Vlontzos A, Glocker B, Kainz B, Rueckert D (2019) Evaluating reinforcement learning agents for anatomical landmark detection. Med Image Anal 53:156–164. https://doi.org/10.1016/j.media.2019.02.007

    Article  PubMed  Google Scholar 

  13. Vlontzos A, Alansary A, Kamnitsas K, Rueckert D, Kainz B (2019) Multiple landmark detection using multi-agent reinforcement learning. In: Shen D et al (eds) Medical image computing and computer assisted intervention—MICCAI 2019, vol 11767. Lecture notes in computer science. Springer, Berlin, pp 262–270. https://doi.org/10.1007/978-3-030-32251-9_29

    Chapter  Google Scholar 

  14. Gauriau R, Cuingnet R, Lesage D, Bloch I (2015) Multi-organ localization with cascaded global-to-local regression and shape prior. Med Image Anal 23(1):70–83. https://doi.org/10.1016/j.media.2015.04.007

    Article  PubMed  Google Scholar 

  15. Criminisi A, Shotton J, Robertson D, Konukoglu E (2010) Regression forests for efficient anatomy detection and localization in CT studies. In: International MICCAI workshop on medical computer vision. Springer, Berlin, pp 106–117. https://doi.org/10.1016/j.media.2013.01.001

  16. Han D, Gao Y, Wu G, Yap PT, Shen D (2014) Robust anatomical landmark detection for MR brain image registration. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8673. Lecture notes in computer science. Springer, Berlin, pp 186–193. https://doi.org/10.1007/978-3-319-10404-1_24

    Chapter  Google Scholar 

  17. Urschler M, Ebner T, Štern D (2018) Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization. Med Image Anal 43:23–36. https://doi.org/10.1016/j.media.2017.09.003

    Article  PubMed  Google Scholar 

  18. Oktay O, Bai W, Guerrero R, Rajchl M, de Marvao A, O’Regan DP, Cook SA, Heinrich MP, Glocker B, Rueckert D (2016) Stratified decision forests for accurate anatomical landmark localization in cardiac images. IEEE Trans Med Imaging 36(1):332–342. https://doi.org/10.1109/TMI.2016.2597270

    Article  Google Scholar 

  19. Zheng Y, Liu D, Georgescu B, Nguyen H, Comaniciu D (2015) 3D deep learning for efficient and robust landmark detection in volumetric data. In: Navab N, Hornegger J, Wells W, Frangi A (eds) Medical image computing and computer-assisted intervention—MICCAI 2015, vol 9349. Lecture notes in computer science. Springer, Berlin, pp 565–572. https://doi.org/10.1007/978-3-319-24553-9_69

    Chapter  Google Scholar 

  20. Bier B, Unberath M, Zaech JN, Fotouhi J, Armand M, Osgood G, Navab N, Maier A (2018) X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery. In: Frangi A, Schnabel J, Davatzikos C, Alberola-López C, Fichtinger G (eds) Medical image computing and computer assisted intervention—MICCAI 2018, vol 11073. Lecture notes in computer science. Springer, Berlin, pp 55–63. https://doi.org/10.1007/978-3-030-00937-3_7

    Chapter  Google Scholar 

  21. Zhang J, Liu M, Shen D (2017) Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks. IEEE Trans Image Process 26(10):4753–4764. https://doi.org/10.1109/TIP.2017.2721106

    Article  PubMed  PubMed Central  Google Scholar 

  22. O’Neil AQ, Kascenas A, Henry J, Wyeth D, Shepherd M, Beveridge E, Clunie L, Sansom C, Seduikyte Keith Muir E, Poole I (2018) Attaining human-level performance with atlas location autocontext for anatomical landmark detection in 3D CT data. In: Leal-Taixé L, Roth S (eds) Computer vision—ECCV 2018 workshops, Lecture notes in computer science, vol 11131. Springer, pp 470–484. https://doi.org/10.1007/978-3-030-11015-4_34

  23. Andermatt S, Pezold S, Amann M, Cattin PC (2017) Multi-dimensional gated recurrent units for automated anatomical landmark localization. arXiv preprint arXiv:1708.02766

  24. Hussain R, Lalande A, Girum KB, Guigou C, Grayeli AB (2019) 3D landmark detection for augmented reality based otologic procedures. In: Surgetica conference, Rennes, France, arXiv preprint arXiv:1909.01647

  25. Payer C, Štern D, Bischof H, Urschler M (2016) Regressing heatmaps for multiple landmark localization using CNNs. In: Ourselin S, Joskowicz L, Sabuncu M, Unal G, Wells W (eds) Medical image computing and computer-assisted intervention—MICCAI 2016, vol 9901. Lecture notes in computer science. Springer, Berlin, pp 230–238. https://doi.org/10.1007/978-3-319-46723-8_27

    Chapter  Google Scholar 

  26. Yang D, Zhang S, Yan Z, Tan C, Li K, Metaxas D (2015) Automated anatomical landmark detection ondistal femur surface using convolutional neural network. In: IEEE 12th international symposium on biomedical imaging (ISBI), pp 17–21. https://doi.org/10.1109/ISBI.2015.7163806

  27. Li Y, Alansary A, Cerrolaza JJ, Khanal B, Sinclair M, Matthew J, Gupta C, Knight C, Kainz B, Rueckert D (2018) Fast multiple landmark localisation using a patch-based iterative network. In: Frangi A, Schnabel J, Davatzikos C, Alberola-López C, Fichtinger G (eds) Medical image computing and computer assisted intervention—MICCAI 2018, vol 11070. Lecture notes in computer science. Springer, Berlin, pp 563–571. https://doi.org/10.1007/978-3-030-00928-1_64

    Chapter  Google Scholar 

  28. Ghesu FC, Georgescu B, Mansi T, Neumann D, Hornegger J, Comaniciu D (2016) An artificial agent for anatomical landmark detection in medical images. In: Ourselin S, Joskowicz L, Sabuncu M, Unal G, Wells W (eds) Medical image computing and computer-assisted intervention—MICCAI 2016, vol 9902. Lecture notes in computer science. Springer, Berlin, pp 229–237. https://doi.org/10.1007/978-3-319-46726-9_27

    Chapter  Google Scholar 

  29. Ghesu FC, Georgescu B, Zheng Y, Grbic S, Maier A, Hornegger J, Comaniciu D (2017) Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans. IEEE Trans Pattern Anal Mach Intell 41(1):176–189. https://doi.org/10.1109/TPAMI.2017.2782687

    Article  PubMed  Google Scholar 

  30. Marroquin R, Lalande A, Hussain R, Guigou C, Grayeli AB (2018) Augmented reality of the middle ear combining otoendoscopy and temporal bone computed tomography. Otol Neurotol 39(8):931–939. https://doi.org/10.1097/MAO.0000000000001922

    Article  PubMed  Google Scholar 

  31. Bland JM, Altman D (1996) Statistics notes: measurement error. BMJ 313:744. https://doi.org/10.1136/bmj.313.7059.744

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. van Dinther J, Zarowski A, Somers T, Offeciers E (2017) The MO-meatocanalplasty: a modification of the M-meatoplasty to address the superior quadrants and the bony canal. Eur Arch Otorhinolaryngol 274(9):3291–3293. https://doi.org/10.1007/s00405-017-4626-4

    Article  PubMed  Google Scholar 

  33. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S (2015) Human-level control through deep reinforcement learning. Nature 518:529–533. https://doi.org/10.1038/nature14236

    Article  CAS  Google Scholar 

  34. Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge

    Google Scholar 

  35. Salvi J, Armangué X, Pagès J (2001) A survey addressing the fundamental matrix estimation problem. IEEE Int Conf Image Process (ICIP) 2:209–212. https://doi.org/10.1109/ICIP.2001.958461

    Article  Google Scholar 

  36. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: European conference on computer vision—ECCV 2006. Lecture notes in computer science, vol 3951. Springer, pp 404–417. https://doi.org/10.1007/11744023_32

  37. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  38. Bozorg Grayeli A, Esquia-Medina G, Nguyen Y, Mazalaigue S, Vellin JF, Lombard B, Kalamarides M, Ferrary E, Sterkers O (2009) Use of anatomic or invasive markers in association with skin surface registration in image-guided surgery of the temporal bone. Acta Otolaryngol 129(4):405–10. https://doi.org/10.1080/00016480802579025

    Article  Google Scholar 

  39. Hussain R, Lalande A, Marroquin R, Guigou C, Grayeli AB (2020) Video-based augmented reality combining CT-scan and instrument position data to microscope view in middle ear surgery. Sci Rep 10:6767. https://doi.org/10.1038/s41598-020-63839-2

Download references

Acknowledgements

The authors would like to thank Oticon Medical, France for their financial support. The authors are also thankful to NVIDIA GPU Grant program for donating the TITAN X processor used in this study.

Author information

Authors and Affiliations

  1. ImViA Laboratory, University of Burgundy, Batiment I3M, 64b Rue Sully, 21000, Dijon, France

    Raabid Hussain, Alain Lalande, Kibrom Berihu Girum, Caroline Guigou & Alexis Bozorg Grayeli

  2. Medical Imaging Department, CHU Dijon, Dijon, France

    Alain Lalande

  3. ENT Department, CHU Dijon, Dijon, France

    Caroline Guigou & Alexis Bozorg Grayeli

Authors
  1. Raabid Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alain Lalande
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kibrom Berihu Girum
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Caroline Guigou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexis Bozorg Grayeli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raabid Hussain.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Informed consent

Ethical approval and informed consent were not required for this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 5416 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hussain, R., Lalande, A., Berihu Girum, K. et al. Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos. Int J CARS 15, 1703–1711 (2020). https://doi.org/10.1007/s11548-020-02240-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02240-w

Keywords

Search

Navigation