Skip to main content
SpringerLink
Log in

Cephalometric Landmark Regression with Convolutional Neural Networks on 3D Computed Tomography Data

  • APPLIED PROBLEMS
  • Published:
Pattern Recognition and Image Analysis Aims and scope Submit manuscript

Abstract

In this paper, we address the problem of automatic three-dimensional cephalometric analysis. Cephalometric analysis performed on lateral radiographs doesn’t fully exploit the structure of 3D objects due to projection onto the lateral plane. With the development of three-dimensional imaging techniques such as CT, several analysis methods have been proposed that extend to the 3D case. The analysis based on these methods is invariant to rotations and translations and can describe difficult skull deformation, where 2D cephalometry has no use. In this paper, we provide a wide overview of existing approaches for cephalometric landmark regression. Moreover, we perform a series of experiments with state of the art 3D convolutional neural network (CNN) based methods for keypoint regression: direct regression with CNN, heatmap regression and Softargmax regression. For the first time, we extensively evaluate the described methods and demonstrate their effectiveness in the estimation of Frankfort Horizontal and cephalometric points locations for patients with severe skull deformations. We demonstrate that Heatmap and Softargmax regression models provide sufficient regression error for medical applications (less than 4 mm). Moreover, the Softargmax model achieves 1.15° inclination error for the Frankfort horizontal. For the fair comparison with the prior art, we also report results projected on the lateral plane.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

Similar content being viewed by others

REFERENCES

  1. A. Jacobson (ed.), Radiographic Cephalometry: From Basics to Videoimaging (Quintessence Publishing, Chicago, 1995).

    Google Scholar 

  2. A. Katsumata, M. Fujishita, M. Maeda, Y. Ariji, E. Ariji, and R. P. Langlais, “3D-CT evaluation of facial asymmetry,” Oral Surg. Oral Med. Oral Pathol. Oral Radiol. Endodontol. 99 (2), 212–220 (2005).

    Article  Google Scholar 

  3. M. G. Cavalcanti, J. W. Haller, and M. W. Vannier, “Three-dimensional computed tomography landmark measurement in craniofacial surgical planning: Experimental validation in vitro,” J. Oral Maxillofac. Surg. 57 (6), 690–694 (1999).

    Article  Google Scholar 

  4. R. Olszewski, F. Zech, G. Cosnard, V. Nicolas, B. Macq, and H. Reychler, “Three-dimensional computed tomography cephalometric craniofacial analysis: Experimental validation in vitro,” Int. J. Oral Maxillofac. Surg. 36 (9), 828–833 (2007).

    Article  Google Scholar 

  5. M. J. Troulis, P. Everett, E. B. Seldin, R. Kikinis, and L. B. Kaban, “Development of a three-dimensional treatment planning system based on computed tomographic data,” Int. J. Oral Maxillofac. Surg. 31, 349–357 (2002).

    Article  Google Scholar 

  6. T. Smektała, M. Jędrzejewski, J. Szyndel, K. Sporniak-Tutak, and R. Olszewski, “Experimental and clinical assessment of three-dimensional cephalometry: A systematic review,” J. Cranio-Maxillofac. Surg. 42 (8), 1795–1801 (2014).

    Article  Google Scholar 

  7. J. Gateno, J. J. Xia, and J. F. Teichgraeber, “New 3-dimensional cephalometric analysis for orthognathic surgery,” J. Oral Maxillofac. Surg. 69 (3), 606–622 (2011).

    Article  Google Scholar 

  8. H.-L. Kim, B. C. Kim, J.-G. Kim, P. Zhengguo, S. H. Kang, and S.-H. Lee, “Construction and validation of the midsagittal reference plane based on the skull base symmetry for three-dimensional cephalometric craniofacial analysis,” J. Craniofac. Surg. 25 (2), 338–342 (2014).

    Article  Google Scholar 

  9. M.-H. Chen, J. Z.-C. Chang, S.-H. Kok, Y.-J. Chen, Y.-D. Huang, K.-Y. Cheng, and C.-P. Lin, “Intraobserver reliability of landmark identification in cone-beam computed tomography-synthesized two-dimensional cephalograms versus conventional cephalometric radiography: A preliminary study,” J. Dent. Sci. 9 (1), 56–62 (2014).

    Article  Google Scholar 

  10. M. B. da Neiva, Á. C. Soares, C. de Oliveira Lisboa, O. de Vasconcellos Vilella, and A. T. Motta, “Evaluation of cephalometric landmark identification on CBCT multiplanar and 3D reconstructions,” Angle Orthod. 85 (1), 11–17 (2015).

    Article  Google Scholar 

  11. D. Grauer, L. S. H. Cevidanes, M. A. Styner, I. Heulfe, E. T. Harmon, H. Zhu, and W. R. Proffit, “Accuracy and landmark error calculation using cone-beam computed tomography-generated cephalograms,” Angle Orthod. 80 (2), 286–294 (2010).

    Article  Google Scholar 

  12. J. J. Xia, J. Gateno, J. F. Teichgraeber, P. Yuan, J. Li, K.-C. Chen, A. Jajoo, M. Nicol, and D. M. Alfi, “Algorithm for planning a double-jaw orthognathic surgery using a computer-aided surgical simulation (CASS) protocol. Part 2: Three-dimensional cephalometry,” Int. J. Oral Maxillofac. Surg. 44 (12), 1441–1450 (2015).

    Article  Google Scholar 

  13. I. Ono, T. Ohura, E. Narumi, K. Kawashima, I. Matsuno, S. Nakamura, N. Ohhata, Y. Uchiyama, Y. Watanabe, F. Tanaka, and T. Kishinami, “Three-dimensional analysis of craniofacial bones using three-dimensional computer tomography,” J. Cranio-Maxillofac. Surg. 20 (2), 49–60 (1992).

    Article  Google Scholar 

  14. I. Hayashi, “Morphological relationship between the cranial base and dentofacial complex obtained by reconstructive computer tomographic images,” Eur. J. Orthod. 25 (4), 385–391 (2003).

    Article  Google Scholar 

  15. B. B. Tuncer, M. S. Ataç, and S. Yüksel, “A case report comparing 3-D evaluation in the diagnosis and treatment planning of hemimandibular hyperplasia with conventional radiography,” J. Cranio-Maxillofac. Surg. 37 (6), 312–319 (2009).

    Article  Google Scholar 

  16. D. Zhang, S. Wang, J. Li, and Y. Zhou, “Novel method of constructing a stable reference frame for 3-dimensional cephalometric analysis,” Am. J. Orthod. Dentofac. Orthop. 154 (3), 397–404 (2018).

    Article  Google Scholar 

  17. O. J. C. van Vlijmen, T. Maal, S. J. Bergé, E. M. Bron-khorst, C. Katsaros, and A. M. Kuijpers-Jagtman, “A comparison between 2D and 3D cephalometry on CBCT scans of human skulls,” Int. J. Oral Maxillofac. Surg. 39 (2), 156–160 (2010).

    Article  Google Scholar 

  18. R. Olszewski, O. Tanesy, G. Cosnard, F. Zech, and H. Reychler, “Reproducibility of osseous landmarks used for computed tomography based three-dimensional cephalometric analyses,” J. Cranio-Maxillofac. Surg. 38 (3), 214–221 (2010).

    Article  Google Scholar 

  19. R. Olszewski, G. Cosnard, B. Macq, P. Mahy, and H. Reychler, “3D CT-based cephalometric analysis: 3D cephalometric theoretical concept and software,” Neuroradiology 48 (11), 853–862 (2006).

    Article  Google Scholar 

  20. G. R. J. Swennen, F. Schutyser, and J.-E. Hausamen, Three-Dimensional Cephalometry. A Color Atlas and Manual (Springer, Berlin, Heidelberg, 2006).

    Book  Google Scholar 

  21. G. J. Edwards, C. J. Taylor, and T. F. Cootes, “Interpreting face images using active appearance models,” in Proc. Third IEEE Int. Conf. on Automatic Face and Gesture Recognition (Nara, Japan, 1998), pp. 300–305.

  22. Y. J. Chen, S. K. Chen, H. F. Chang, and K. C. Chen, “Comparison of landmark identification in traditional versus computer-aided digital cephalometry,” Angle Orthod. 70 (5), 387–392 (2000).

    Google Scholar 

  23. I. El-Feghi, M. A. Sid-Ahmed, and M. Ahmadi, “Automatic localization of craniofacial landmarks for assisted cephalometry,” in Proc. 2003 Int. Symposium on Circuits and Systems (ISCAS 2003) (Bangkok, Thailand, 2003), IEEE, Vol. 3, pp. III-630–III-633; Pattern Recogn. 37 (3), 609–621 (2004).

  24. A. A. Pouyan and M. Farshbaf, “Cephalometric landmarks localization based on histograms of oriented gradients,” in Proc. 2010 Int. Conf. on Signal and Image Processing (ICSIP 2010) (Chennai, India, 2010), IEEE, pp. 1–6.

  25. M. Dantone, J. Gall, G. Fanelli, and L. Van Gool, “Real-time facial feature detection using conditional regression forests,” in Proc. 2012 IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI, 2012), pp. 2578–2585.

  26. M. Osadchy, Y. Le Cun, and M. L. Miller, “Synergistic face detection and pose estimation with energy-based models,” in Toward Category-Level Object Recognition, Ed. by J. Ponce, M. Hebert, C. Schmid, and A. Zisserman, Lecture Notes in Computer Science (Springer, Berlin, Heidelberg, 2006), Vol. 4170, pp. 196–206.

    Google Scholar 

  27. J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, “Efficient object localization using Convolutional Networks,” in Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Boston, MA, 2015), pp. 648–656.

  28. A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in Computer Vision–ECCV 2016, Proc. 14th European Conf., Part VIII, Ed. by B. Leibe et al., Lecture Notes in Computer Science (Springer, Cham, 2016), Vol. 9912, pp. 483–499.

  29. A. Nibali, Z. He, S. Morgan, and L. Prendergast, “Numerical coordinate regression with convolutional neural networks,” arXiv preprint arXiv:1801.07372 (2018). https://arxiv.org/abs/1801.07372

  30. W. Yue, D. Yin, C. Li, G. Wang, and T. Xu, “Automated 2-D cephalometric analysis on X-ray images by a model-based approach,” IEEE Trans. Biomed. Eng. 53 (8), 1615– 1623 (2006).

    Article  Google Scholar 

  31. D. A. Lachinov, A. A. Getmanskaya, and V. E. Turlapov, “Refinement of the coherent point drift registration results by the example of cephalometry problems,” Program. Comput. Software 44 (4), 248–257 (2018).

    Article  Google Scholar 

  32. A. Myronenko and X. B. Song, “Point set registration: Coherent point drift,” IEEE Trans. Pattern Anal. Mach. Intell. 32 (12), 2262–2275 (2010).

    Article  Google Scholar 

  33. C. Lee, C. Tanikawa, J.-Y. Lim, and T. Yamashiro, “Deep learning based cephalometric landmark identification using landmark-dependent multi-scale patches,” arXiv preprint arXiv:1906.02961 (2019). https://arxiv.org/abs/1906.02961

  34. R. Chen, Y. Ma, N. Chen, D. Lee, and W. Wang, “Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2019, Ed. by D. Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, and A. Khan, Lecture Notes in Computer Science (Springer, Cham, 2019), Vol. 11766, pp. 873–881.

    Google Scholar 

  35. H.-W. Hwang, J.-H. Park, J.-H. Moon, Y. Yu, H. Kim, S.-B. Her, G. Srinivasan, M. N. A. Aljanabi, R. E. Donatelli, and S.-J. Lee, “Automated identification of cephalometric landmarks: Part 2–Might it be better than human?” Angle Orthod. 90 (1), 69–76 (2020).

    Article  Google Scholar 

  36. J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv:1804.02767 (2018). https://arxiv.org/abs/1804.02767

  37. S. M. Lee, H. P. Kim, K. Jeon, S.-H. Lee, and J. K. Seo, “Automatic 3D cephalometric annotation system using shadowed 2D image-based machine learning,” Phys. Med. Biol. 64 (5), 055002 (2019).

    Article  Google Scholar 

  38. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 (2014). https://arxiv.org/abs/1409.1556

  39. S. H. Kang, K. Jeon, H.-J. Kim, J. K. Seo, and S.‑H. Lee, “Automatic three-dimensional cephalometric annotation system using three-dimensional convolutional neural networks: A developmental trial,” Comput. Methods Biomech. Biomed. Eng.: Imaging & Visualization 8 (2), 210–218 (2020).

    Google Scholar 

  40. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

  41. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Proc. 2017 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017) (Honolulu, HI, 2017), pp. 5987–5995.

  42. Y. Wu and K. He, “Group normalization,” in Computer Vision−ECCV 2018, Proc. 15th European Conference, Part XIII, Ed. by V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Lecture Notes in Computer Science (Springer, Cham, 2018), Vol. 11217, pp. 3–19; Int. J. Comput. Vision 128 (3), 742–755 (2020).

  43. D. C. Luvizon, H. Tabia, and D. Picard, “Human pose regression by combining indirect part detection and contextual information,” Comput. Graphics 85, 15–22 (2017).

    Article  Google Scholar 

  44. C. Goch, J. Metzger, and M. Nolden, “Tutorial: Medical image processing with MITK introduction and new developments,” in Bildverarbeitung für die Medizin 2017, Ed. by K. H. Majer-Hein, T. M. Deserno, H. Handels, and T. Tolxdorff, Informatik aktuell. Springer, Berlin, Heidelberg, 2017), p. 10. http://mitk.org/wiki/MITK

  45. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32: Proc. NIPS 2019 Conf. (Vancouver, Canada, 2019), pp. 8024–8035.

Download references

ACKNOWLEDGMENTS

This work was supported by the Russian Foundation for Basic Research, project no. 18-37-00383.

Author information

Authors and Affiliations

  1. Institute of Information Technologies, Mathematics and Mechanics, Lobachevsky State University, Nizhny Novgorod, Russia

    D. Lachinov, A. Getmanskaya & V. Turlapov

  2. Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria

    D. Lachinov

Authors
  1. D. Lachinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Getmanskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. V. Turlapov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to D. Lachinov, A. Getmanskaya or V. Turlapov.

Ethics declarations

The authors declare they have no conflicts of interest.

Additional information

Dmitrii Lachinov. Born in 1993. MSc degree received from Lobachevsky State University, Institute of Information Technologies, Mathematics, and Mechanics in 2018. In the 2018–2019 academic year, he was a graduate student at Lobachevsky University Currently pursuing a PhD degree at the Medical University of Vienna, Department of Ophthalmology and Optometry. Research Assistant at the Medical University of Vienna. Area of research: medical imaging and image processing.

Aleksandra Getmanskaya. Born in 1986. MSc degree received from Lobachevsky State University, Institute of Information Technologies, Mathematics, and Mechanics in 2009. Currently is a researcher at Lobachevsky State University, Institute of Information Technologies, Mathematics and Mechanics. Area of research: medical imaging and image processing.

Vadim Turlapov. Born in 1949. In 1994, he received a PhD degree, and in 2002, Doctor of Science, in the field of engineering geometry and computer graphics. Currently is a professor of Lobachevsky State University, head of Computer Graphics Lab in the Institute of Information Technologies, Mathematics, and Mechanics. Area of research: scientific visualization, medical imaging, image processing, hyperspectral imaging, global illumination, photorealistic image synthesis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lachinov, D., Getmanskaya, A. & Turlapov, V. Cephalometric Landmark Regression with Convolutional Neural Networks on 3D Computed Tomography Data. Pattern Recognit. Image Anal. 30, 512–522 (2020). https://doi.org/10.1134/S1054661820030165

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1054661820030165

Keywords:

Search

Navigation