Skip to main content
SpringerLink
Log in

A novel method for 3D knee anatomical landmark localization by combining global and local features

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Landmark localization with neural networks had gained popularity in recent years. However, due to the high dimensionality and large size of medical images, current neural network models still have problems such as information loss with deeper network, low accuracy and robustness. To address these issues, a 3D anatomical landmark localization method with a two-stage strategy was proposed in this study. The 3D spatial information between landmarks and the local information of each single feature point were extracted in these two stages. Additionally, new inception and attention modules were designed for the second stage to combine convolutional kernels of different sizes and weight labeling to strengthen the effective features extraction while weakening the invalid features. The proposed model was evaluated on a collected knee image dataset. The results outperformed state-of-the-art models with a mean error of 3.29 mm and a standard deviation of 2.17 mm. The outlier rates at error radius 3 mm, 5 mm and 7 mm were 53%, 22% and 5%, respectively, indicating good robustness of the model. The study provides a new neural network model with good accuracy for landmark localization tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Beichel, R., Bischof, H., Leberl, F., Sonka, M.: Robust active appearance models and their application to medical image analysis. IEEE Trans. Med. Imaging 24(9), 1151–1169 (2005)

    Article  Google Scholar 

  2. Frisoni, G.B., Fox, N.C., Jack, C.R., Jr., Scheltens, P., Thompson, P.M.: The clinical use of structural MRI in Alzheimer disease. Nat. Rev. Neurol. 6(2), 67–77 (2010)

    Article  Google Scholar 

  3. Jaramaz, B., Hafez, M.A., DiGioia, A.M.: Computer-assisted orthopaedic surgery. Proc. IEEE 94(9), 1689–1695 (2006)

    Article  Google Scholar 

  4. Zheng, G., Nolte, L.P.: Computer-assisted orthopedic surgery: current state and future perspective. Front. Surg. 2(66), 66 (2015)

    Google Scholar 

  5. Yu, F., Li, L., Teng, H.J., Shi, D.Q., Jiang, Q.: Robots in orthopedic surgery. Ann. Jt. 3(3), 15–15 (2018)

    Article  Google Scholar 

  6. Victor, J., Van Doninck, D., Labey, L., Innocenti, B., Parizel, P.M., Bellemans, J.: How precise can bony landmarks be determined on a CT scan of the knee? Knee 16(5), 358–365 (2009)

    Article  Google Scholar 

  7. Zhang, J., Liu, M., Le, A., Gao, Y., Shen, D.: Alzheimer’s disease diagnosis using landmark-based features from longitudinal structural MR images. IEEE J. Biomed. Health Inform. 21(6), 1607–1616 (2017)

    Article  Google Scholar 

  8. Gan K.: Automated Localization of Anatomical Landmark Points in 3D Medical Images. In: IEEE International Conference on Digital Signal Processing (Dsp), pp. 143–147.

  9. Ashburner, J., Friston, K.J.: Voxel-based morphometry–the methods. Neuroimage 11(6 Pt 1), 805–821 (2000)

    Article  Google Scholar 

  10. Tiulpin, A., Melekhov, I., Saarakkala, S.: KNEEL: knee anatomical landmark localization using hourglass networks. IEEE Int. Conf. Comp. V 1, 352–361 (2019)

    Google Scholar 

  11. Liu, C.B., Xie, H.T., Zhang, S.C., Mao, Z.D., Sun, J., Zhang, Y.D.: Misshapen Pelvis landmark detection with local-global feature learning for diagnosing developmental dysplasia of the hip. IEEE Trans. Med. Imaging 39(12), 3944–3954 (2020)

    Article  Google Scholar 

  12. Xu, J., Xie, H., Liu, C., Yang, F., Zhang, S., Chen, X., Zhang, Y.: Hip Landmark detection with dependency mining in ultrasound image. IEEE Trans. Med. Imaging 40(12), 3762–3774 (2021)

    Article  Google Scholar 

  13. Payer, C., Stern, D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based CNNs for landmark localization. Med. Image Anal. 54, 207–219 (2019)

    Article  Google Scholar 

  14. Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., Quan, L.: Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5520–5529.

  15. Fritscher, K., Raudaschl, P., Zaffino, P., Spadea, M., Sharp, G.: Deep Neural Networks for Fast Segmentation of 3D Medical Images (2016).

  16. Zhao, N.N., Tong, N., Ruan, D., Sheng, K.: Fully automated pancreas segmentation with two-stage 3D convolutional neural networks. Med. Image Comput. Comput. Assist. Interv. - Miccai 2019 Pt Ii 11765, 201–209 (2019)

    Google Scholar 

  17. O’Neil, A.Q., Kascenas, A., Henry, J., Wyeth, D., Shepherd, M., Beveridge, E., Clunie, L., Sansom, C., Šeduikytė, E., Muir, K., Poole, I.: Attaining human-level performance with atlas location autocontext for anatomical landmark detection in 3D CT data. Computer Vision – ECCV 2018 Workshops, pp. 470–484.

  18. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.H.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  19. Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing Heatmaps for Multiple Landmark Localization Using CNNs. Springer International Publishing, Cham, pp. 230–238 (2016).

  20. Zhang, J., Liu, M., Shen, D.: Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks. IEEE Trans. Image Process. 26(10), 4753–4764 (2017)

    Article  MathSciNet  Google Scholar 

  21. Qian, J. H., Cheng, M., Tao, Y. B., Lin, J., and Lin, H.: CephaNet: An Improved Faster R-CNN for Cephalometric Landmark Detection. In: IEEE 16th International Symposium on Biomedical Imaging (Isbi 2019), pp. 868–871 (2019).

  22. Urschler, M., Ebner, T., Stern, D.: Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization. Med. Image Anal. 43, 23–36 (2018)

    Article  Google Scholar 

  23. Pfister, T., Charles, J., Zisserman, A.: Flowing ConvNets for human pose estimation in videos. IEEE I Conf Comp Vis 1, 1913–1921 (2015)

    Google Scholar 

  24. Zhou, S.K.: Discriminative anatomy detection: Classification vs regression. Pattern Recogn. Lett. 43, 25–38 (2014)

    Article  Google Scholar 

  25. Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J., van Ginneken, B., Sanchez, C.I.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)

    Article  Google Scholar 

  26. Torosdagli, N., Liberton, D.K., Verma, P., Sincan, M., Lee, J.S., Bagci, U.: Deep geodesic learning for segmentation and anatomical landmarking. IEEE Trans. Med. Imaging 38(4), 919–931 (2019)

    Article  Google Scholar 

  27. Wolterink, J.M., van Hamersvelt, R.W., Viergever, M.A., Leiner, T., Isgum, I.: Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier. Med. Image Anal. 51, 46–60 (2019)

    Article  Google Scholar 

  28. Zhang, D., Wang, J., Noble, J.H., Dawant, B.M.: HeadLocNet: Deep convolutional neural networks for accurate classification and multi-landmark localization of head CTs. Med. Image Anal. 61, 1059 (2020)

    Article  Google Scholar 

  29. Newell, A., Yang, K., Deng, J.: Stacked Hourglass Networks for Human Pose Estimation, pp. 483–499. Springer, Cham (2016)

    Google Scholar 

  30. Zheng, Y., Liu, D., Georgescu, B., Nguyen, H., Comaniciu, D.: 3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data, pp. 565–572. Springer, Cham (2015)

    Google Scholar 

  31. Riegler, G., Urschler, M., Ruther, M., Bischof, H., Stern, D.: Anatomical landmark detection in medical applications driven by synthetic data. In: 2015 Ieee International Conference on Computer Vision Workshop (Iccvw), pp. 85–89 (2015).

  32. Liao, H., Mesfin, A., Luo, J.: Joint Vertebrae Identification and Localization in Spinal CT Images by Combining Short- and Long-Range Contextual Information. IEEE Trans. Med. Imaging 37(5), 1266–1275 (2018)

    Article  Google Scholar 

  33. Li, Y., Alansary, A., Cerrolaza, J., Khanal, B., Sinclair, M., Matthew, J., Gupta, C., Knight, C., Kainz, B., Rueckert, D.: Fast Multiple Landmark Localisation Using a Patch-based Iterative Network (2018).

  34. Noothout, J.M.H., De Vos, B.D., Wolterink, J.M., Postma, E.M., Smeets, P.A.M., Takx, R.A.P., Leiner, T., Viergever, M.A., Isgum, I.: Deep learning-based regression and classification for automatic landmark localization in medical images. IEEE Trans. Med. Imaging 39(12), 4011–4022 (2020)

    Article  Google Scholar 

  35. Imran, A.-A.-Z., Huang, C., Tang, H., Cheung, K., To, M., Qian, Z., Terzopoulos, D.: Bipartite Distance for Shape-Aware Landmark Detection in Spinal X-Ray Images (2020).

  36. Miotto, R., Wang, F., Wang, S., Jiang, X., Dudley, J.T.: Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform. 19(6), 1236–1246 (2018)

    Article  Google Scholar 

  37. Zhao, Q., Zhu, J., Zhu, J., Zhou, A., Shao, H.: Bone anatomical landmark localization with cascaded spatial configuration network. Measur. Sci. Technol. 33(6), 065401 (2022)

    Article  Google Scholar 

  38. Zhang, Z.W., Mao, S.T., Coyle, J., Sejdic, E.: Automatic annotation of cervical vertebrae in videofluoroscopy images via deep learning. Med. Image Anal. 74, 1 (2021)

    Google Scholar 

  39. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition (2015). CoRR, abs/1409.1556.

  40. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.A.: Striving for Simplicity: The All Convolutional Net (2015) CoRR, abs/1412.6806.

  41. Seo, H., Huang, C., Bassenne, M., Xiao, R., Xing, L.: Modified U-Net (mU-Net) with incorporation of object-dependent high level features for improved liver and liver-tumor segmentation in CT images. IEEE Trans. Med. Imaging 39(5), 1316–1325 (2020)

    Article  Google Scholar 

  42. Shi, J., Li, Z., Ying, S., Wang, C., Liu, Q., Zhang, Q., Yan, P.: MR image super-resolution via wide residual networks with fixed skip connection. IEEE J. Biomed. Health Inform. 23(3), 1129–1140 (2019)

    Article  Google Scholar 

  43. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation, pp. 234–241. Springer, Cham (2015)

    Google Scholar 

  44. Szegedy, C., Liu, W., Jia, Y. Q., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going Deeper with Convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (Cvpr), pp. 1–9 (2015).

  45. van der Maaten, L., Postma, E., Herik, H.: Dimensionality reduction: a comparative review. J. Mach. Learn. Res. - JMLR 10, 1 (2007)

    Google Scholar 

  46. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the Inception Architecture for Computer Vision. Proc Cvpr Ieee, pp. 2818–2826 (2016).

  47. Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. ArXiv, 1409 (2014).

  48. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I.: Attention Is All You Need. Adv Neur In, 30 (2017).

  49. Wu, Y.X., He, K.M.: Group normalization. Comput. Vis. - Eccv 2018 Pt Xiii 11217, 3–19 (2018)

    Article  Google Scholar 

  50. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15, 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  51. Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. Int Conf 3d Vision, pp. 565–571 (2016).

  52. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. Springer International Publishing, Cham (2016)

    Google Scholar 

  53. Milletari, F., Navab, N., Ahmadi, S.: V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571 (2016).

Download references

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China

    Junjun Zhu, Qijie Zhao & Junhao Zhu

  2. MicroPort Medbot (Group) Co., Ltd., Shanghai, China

    Anwen Zhou & Hui Shao

Authors
  1. Junjun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qijie Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junhao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anwen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qijie Zhao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, J., Zhao, Q., Zhu, J. et al. A novel method for 3D knee anatomical landmark localization by combining global and local features. Machine Vision and Applications 33, 52 (2022). https://doi.org/10.1007/s00138-022-01303-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-022-01303-z

Keywords

Search

Navigation