Skip to main content
Log in

An efficient brain tumor image classifier by combining multi-pathway cascaded deep neural network and handcrafted features in MR images

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Accurate segmentation and delineation of the sub-tumor regions are very challenging tasks due to the nature of the tumor. Traditionally, convolutional neural networks (CNNs) have succeeded in achieving most promising performance for the segmentation of brain tumor; however, handcrafted features remain very important in identification of tumor’s boundary regions accurately. The present work proposes a robust deep learning–based model with three different CNN architectures along with pre-defined handcrafted features for brain tumor segmentation, mainly to find out more prominent boundaries of the core and enhanced tumor regions. Generally, automatic CNN architecture does not use the pre-defined handcrafted features because it extracts the features automatically. In this present work, several pre-defined handcrafted features are computed from four MRI modalities (T2, FLAIR, T1c, and T1) with the help of additional handcrafted masks according to user interest and fed to the convolutional features (automatic features) to improve the overall performance of the proposed CNN model for tumor segmentation. Multi-pathway CNN is explored in this present work along with single-pathway CNN, which extracts simultaneously both local and global features to identify the accurate sub-regions of the tumor with the help of handcrafted features. The present work uses a cascaded CNN architecture, where the outcome of a CNN is considered as an additional input information to next subsequent CNNs. To extract the handcrafted features, convolutional operation was applied on the four MRI modalities with the help of several pre-defined masks to produce a predefined set of handcrafted features. The present work also investigates the usefulness of intensity normalization and data augmentation in pre-processing stage in order to handle the difficulties related to the imbalance of tumor labels. The proposed method is experimented on the BraST 2018 datasets and achieved promising results than the existing (currently published) methods with respect to different metrics such as specificity, sensitivity, and dice similarity coefficient (DSC) for complete, core, and enhanced tumor regions. Quantitatively, a notable gain is achieved around the boundaries of the sub-tumor regions using the proposed two-pathway CNN along with the handcrafted features.

This data is mandatory. Please provide.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

References

  1. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R et al (2015) The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  2. Bauer S, Wiest R, Nolte L. -P., Reyes M (2013) A survey of mri-based medical image analysis for brain tumor studies. Phys Med Biol 58(13):R97

    Article  Google Scholar 

  3. Bal A, Banerjee M, Sharma P, Maitra M (2020) Gray matter segmentation and delineation from positron emission tomography (pet) image. In: Emerging technology in modelling and graphics. Springer, pp 359–372

  4. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R et al (2014) The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  5. Lyksborg M, Puonti O, Agn M, Larsen R (2015) An ensemble of 2d convolutional neural networks for tumor segmentation. In: Scandinavian conference on image analysis. Springer, pp 201–211

  6. Kleesiek J, Biller A, Urban G, Kothe U, Bendszus M, Hamprecht F (2014) Ilastik for multi-modal brain tumor segmentation. In: Proceedings MICCAI BraTS (Brain tumor segmentation challenge), pp 12–17

  7. Dvořák P, Menze B (2015) Local structure prediction with convolutional neural networks for multimodal brain tumor segmentation. In: International MICCAI workshop on medical computer vision. Springer, pp 59–71

  8. Bal A, Banerjee M, Chakrabarti A, Sharma P (2018) Mri brain tumor segmentation and analysis using rough-fuzzy c-means and shape based properties. Journal of King Saud University-Computer and Information Sciences

  9. Bal A, Banerjee M, Sharma P, Maitra M (2018) Brain tumor segmentation on mr image using k-means and fuzzy-possibilistic clustering. In: 2018 2nd international conference on electronics, materials engineering & nano-technology (IEMENTech). IEEE, pp 1–8

  10. Bal A, Banerjee M, Sharma P, Chaki R (2020) A multi-class image classifier for assisting in tumor detection of brain using deep convolutional neural network. In: Advanced computing and systems for security. Springer, pp 93–111

  11. Hussain S, Anwar SM, Majid M (2017) Brain tumor segmentation using cascaded deep convolutional neural network. In: 2017 39th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 1998– 2001

  12. Pereira S, Pinto A, Alves V, Silva CA (2016) Brain tumor segmentation using convolutional neural networks in mri images. IEEE Trans Med Imaging 35(5):1240–1251

    Article  Google Scholar 

  13. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin P-M, Larochelle H (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31

    Article  Google Scholar 

  14. Taheri S, Ong SH, Chong V (2010) Level-set segmentation of brain tumors using a threshold-based speed function. Image Vis Comput 28(1):26–37

    Article  Google Scholar 

  15. Havaei M, Larochelle H, Poulin P, Jodoin P-M (2016) Within-brain classification for brain tumor segmentation. Int J Comput Assist Radiol Surg 11(5):777–788

    Article  Google Scholar 

  16. Georgiadis P, Cavouras D, Kalatzis I, Daskalakis A, Kagadis GC, Sifaki K, Malamas M, Nikiforidis G, Solomou E (2008) Improving brain tumor characterization on mri by probabilistic neural networks and non-linear transformation of textural features. Comput Methods Programs Biomed 89(1):24–32

    Article  Google Scholar 

  17. Wu M-N, Lin C-C, Chang C-C (2007) Brain tumor detection using color-based k-means clustering segmentation. In: Third international conference on intelligent information hiding and multimedia signal processing (IIH-MSP 2007), vol 2. IEEE, pp 245–250

  18. Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS (1998) Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imaging 17(2):187– 201

    Article  CAS  Google Scholar 

  19. Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Automatic brain tumor detection and segmentation using u-net based fully convolutional networks. In: Annual conference on medical image understanding and analysis. Springer, pp 506–517

  20. Sompong C, Wongthanavasu S (2016) Brain tumor segmentation using cellular automata-based fuzzy c-means. In: Computer science and software engineering (JCSSE), 2016 13th international joint conference on. IEEE, pp 1–6

  21. Kwon D, Shinohara RT, Akbari H, Davatzikos C (2014) Combining generative models for multifocal glioma segmentation and registration. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 763–770

  22. Tustison NJ, Shrinidhi K, Wintermark M, Durst CR, Kandel BM, Gee JC, Grossman MC, Avants BB (2015) Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with antsr. Neuroinformatics 13(2):209–225

    Article  Google Scholar 

  23. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Machine Intell 40(4):834–848

    Article  Google Scholar 

  24. Casamitjana A, Puch S, Aduriz A, Sayrol E, Vilaplana V (2016) 3d convolutional networks for brain tumor segmentation. In: Proceedings of the MICCAI challenge on multimodal brain tumor image segmentation (BRATS), pp 65–68

  25. McKinley R, Meier R, Wiest R (2018) Ensembles of densely-connected cnns with label-uncertainty for brain tumor segmentation. In: International MICCAI brainlesion workshop. Springer, pp 456–465

  26. Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med Image Anal 36:61–78

    Article  Google Scholar 

  27. Hussain S, Anwar SM, Majid M (2018) Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing 282:248–261

    Article  Google Scholar 

  28. Chaplot S, Patnaik L, Jagannathan N (2006) Classification of magnetic resonance brain images using wavelets as input to support vector machine and neural network. Biomed Signal Process Control 1 (1):86–92

    Article  Google Scholar 

  29. Pawlak Z (2012) Rough sets: Theoretical aspects of reasoning about data, vol 9. Springer Science & Business Media, New York

    Google Scholar 

  30. Saha R, Phophalia A, Mitra SK (2016) Brain tumor segmentation from multimodal mr images using rough sets. In: International conference on computer vision Graphics, and Image processing. Springer, pp 133–144

  31. Maji P, Pal SK (2007) Rough set based generalized fuzzy c-means algorithm and quantitative indices. IEEE Trans Syst Man Cybern Part B (Cybernetics) 37(6):1529–1540

    Article  Google Scholar 

  32. Maji P, Pal SK (2008) Maximum class separability for rough-fuzzy c-means based brain mr image segmentation. Trans Rough Sets 9:114–134

    Google Scholar 

  33. Hamamci A, Kucuk N, Karaman K, Engin K, Unal G (2012) Tumor-cut: Segmentation of brain tumors on contrast enhanced mr images for radiosurgery applications. IEEE Trans Med Imaging 31 (3):790–804

    Article  Google Scholar 

  34. Görlitz L, Menze B, Weber M-A, Kelm B, Hamprecht F (2007) Semi-supervised tumor detection in magnetic resonance spectroscopic images using discriminative random fields. Pattern Recogn 224–233

  35. Mohammed ZF, Abdulla AA (2020) Thresholding-based white blood cells segmentation from microscopic blood images. UHD J Sci Technol 4(1):9–17

    Article  Google Scholar 

  36. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. nature 521(7553):436

    Article  CAS  Google Scholar 

  37. Khan H, Shah PM, Shah MA, ul Islam S, Rodrigues JJ (2020) Cascading handcrafted features and convolutional neural network for iot-enabled brain tumor segmentation, Computer Communications

  38. Bao P, Zhang L, Wu X (2005) Canny edge detection enhancement by scale multiplication. IEEE Trans Pattern Anal Machine Intell 27(9):1485–1490

    Article  Google Scholar 

  39. Gonzalez CI, Melin P, Castro JR, Mendoza O, Castillo O (2016) An improved sobel edge detection method based on generalized type-2 fuzzy logic. Soft Comput 20(2):773–784

    Article  Google Scholar 

  40. Shrivakshan G, Chandrasekar C (2012) A comparison of various edge detection techniques used in image processing. Int J Comput Sci Issues (IJCSI) 9(5):269

    Google Scholar 

  41. Johnson R (1990) Contrast based edge detection. Pattern Recognit 23(3-4):311–318

    Article  Google Scholar 

  42. Liu L, Zhao L, Long Y, Kuang G, Fieguth P (2012) Extended local binary patterns for texture classification. Image Vis Comput 30(2):86–99

    Article  Google Scholar 

  43. Ilonen J, Kamarainen J-K, Paalanen P, Hamouz M, Kittler J, Kalviainen H (2008) Image feature localization by multiple hypothesis testing of gabor features. IEEE Trans Image Process 17(3):311–325

    Article  Google Scholar 

  44. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense sift. Information Fusion 23:139–155

    Article  Google Scholar 

  45. Wang G, Li W, Vercauteren T, Ourselin S (2019) Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation. Front Computat Neurosci 13:56

    Article  Google Scholar 

  46. Myronenko A, Hatamizadeh A (2020) Robust semantic segmentation of brain tumor regions from 3d mris. In: International MICCAI brainlesion workshop. Springer, pp 82–89

  47. Beers A, Chang K, Brown J, Sartor E, Mammen C, Gerstner E, Rosen B, Kalpathy-cramer J (2020) Sequential 3d u-nets for biologically-informed brain tumor segmentation, arXiv:1709.02967

  48. Shen L, Anderson T (2020) Multimodal brain mri tumor segmentation via convolutional neural networks

  49. Wu W, Li D, Du J, Gao X, Gu W, Zhao F, Feng X, Yan H (2020) An intelligent diagnosis method of brain mri tumor segmentation using deep convolutional neural network and svm algorithm. Computational and Mathematical Methods in Medicine

  50. Zeineldin RA, Karar ME, Coburger J, Wirtz CR, Burgert O (2020) Deepseg: Deep neural network framework for automatic brain tumor segmentation using magnetic resonance flair images, International Journal Of Computer Assisted Radiology And Surgery

  51. Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH (2017) Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge. In: International MICCAI brainlesion workshop. Springer, pp 287–297

  52. Soltaninejad M, Zhang L, Lambrou T, Allinson N, Ye X (2020) Multimodal mri brain tumor segmentation using random forests with features learned from fully convolutional neural network, arXiv:1704.08134

  53. Zikic D, Ioannou Y, Brown M, Criminisi A (2014) Segmentation of brain tumor tissues with convolutional neural networks. Proceedings MICCAI-BRATS 36–39

  54. Li C, Gore JC, Davatzikos C (2014) Multiplicative intrinsic component optimization (mico) for mri bias field estimation and tissue segmentation. Magnetic Resonance Imaging 32(7):913–923

    Article  CAS  Google Scholar 

  55. Nyúl LG, Udupa JK, Zhang X (2000) New variants of a method of mri scale standardization. IEEE Trans Med Imaging 19(2):143–150

    Article  Google Scholar 

  56. Bal A, Banerjee M, Sharma P, Maitra M (2019) An efficient wavelet and curvelet-based pet image denoising technique. Med Biol Eng Comput 57(12):2567–2598

    Article  Google Scholar 

  57. Bal A, Banerjee M, Chaki R, Sharma P (2020) An efficient method for pet image denoising by combining multi-scale transform and non-local means. Multimed Tools Appl 79(39):29087–29120

    Article  Google Scholar 

  58. Wells WM, Grimson WEL, Kikinis R, Jolesz FA (1996) Adaptive segmentation of mri data. IEEE Trans Med Imaging 15(4):429–442

    Article  CAS  Google Scholar 

  59. Van Leemput K, Maes F, Vandermeulen D, Suetens P (1999) Automated model-based bias field correction of mr images of the brain. IEEE Trans Med Imaging 18(10):885–896

    Article  Google Scholar 

  60. Pham DL, Prince JL (1999) Adaptive fuzzy segmentation of magnetic resonance images. IEEE Trans Med Imaging 18(9):737–752

    Article  CAS  Google Scholar 

  61. Ansari MA, Kurchaniya D, Dixit M (2017) A comprehensive analysis of image edge detection techniques. Int J Multimed Ubiquit Eng 12(11):1–12

    Article  Google Scholar 

  62. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110

    Article  Google Scholar 

  63. Wong SC, Gatt A, Stamatescu V, McDonnell MD (2016) Understanding data augmentation for classification: when to warp?. In: Digital image computing: Techniques and applications (DICTA), 2016 international conference on. IEEE, pp 1–6

  64. Fawzi A, Samulowitz H, Turaga D, Frossard P (2016) Adaptive data augmentation for image classification. In: Image processing (ICIP), 2016 IEEE international conference on. Ieee, pp 3688–3692

  65. Wang J, Perez L (2017) The effectiveness of data augmentation in image classification using deep learning. Tech. rep. Technical report

  66. Urban G, Bendszus M, Hamprecht F, Kleesiek J (2014) Multi-modal brain tumor segmentation using deep convolutional neural networks. In: MICCAI braTS (Brain Tumor Segmentation) Challenge Proceedings, winning contribution, pp 31–35

  67. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256

  68. Ye C, Zhao C, Yang Y, Fermüller C, Aloimonos Y (2016) Lightnet: a versatile, standalone matlab-based environment for deep learning. In: Proceedings of the 24th ACM international conference on multimedia. ACM, pp 1156–1159

  69. Mallat S (2009) A wavelet tour of signal processing: The sparse way 3th edn

  70. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  71. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. J Machine Learn Res 15(1):1929–1958

    Google Scholar 

  72. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  73. Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, vol 30, p 3

  74. Zhao L, Jia K (2016) Multiscale cnns for brain tumor segmentation and diagnosis, Computational and mathematical methods in medicine

  75. Gonzales RC, Woods RE (2002) Digital image processing

  76. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, Freymann JB, Farahani K, Davatzikos C (2017) Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data 4:170117

    Article  Google Scholar 

  77. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J, Freymann J, Farahani K, Davatzikos C (2020) Segmentation labels and radiomic features for the pre-operative scans of the tcga-lgg collection, The Cancer Imaging Archive 286

Download references

Acknowledgements

The current work was carried out with an inspirational support from the Board of Research in Nuclear Sciences (ref no. 34/14/13/2016-BRNS/34044). Heartiest thanks to Dr. Punit Sharma for providing the precious suggestions and validating the results as a neuro-consultant at Apollo Gleneagles Hospital, Kolkata, India.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Abhishek Bal.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bal, A., Banerjee, M., Chaki, R. et al. An efficient brain tumor image classifier by combining multi-pathway cascaded deep neural network and handcrafted features in MR images. Med Biol Eng Comput 59, 1495–1527 (2021). https://doi.org/10.1007/s11517-021-02370-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-021-02370-6

Keywords

Navigation