Abstract
Counting the mitotic cells in histopathological cancerous tissue areas is the most relevant indicator of tumor grade in aggressive breast cancer diagnosis. In this paper, we propose a robust and accurate technique for the automatic detection of mitoses from histological breast cancer slides using the multi-task deep learning framework for object detection and instance segmentation Mask RCNN. Our mitosis detection and instance segmentation framework is deployed for two main tasks: it is used as a detection network to perform mitosis localization and classification in the fully annotated mitosis datasets (i.e., the pixel-level annotated datasets), and it is used as a segmentation network to estimate the mitosis mask labels for the weakly annotated mitosis datasets (i.e., the datasets with centroid-pixel labels only). We evaluate our approach on the fully annotated 2012 ICPR grand challenge dataset and the weakly annotated 2014 ICPR MITOS-ATYPIA challenge dataset. Our evaluation experiments show that we can obtain the highest F-score of 0.863 on the 2012 ICPR dataset by applying the mitosis detection and instance segmentation model trained on the pixel-level labels provided by this dataset. For the weakly annotated 2014 ICPR dataset, we first employ the mitosis detection and instance segmentation model trained on the fully annotated 2012 ICPR dataset to segment the centroid-pixel annotated mitosis ground truths, and produce the mitosis mask and bounding box labels. These estimated labels are then used to train another mitosis detection and instance segmentation model for mitosis detection on the 2014 ICPR dataset. By adopting this two-stage framework, our method outperforms all state-of-the-art mitosis detection approaches on the 2014 ICPR dataset by achieving an F-score of 0.475. Moreover, we show that the proposed framework can also perform unsupervised mitosis detection through the estimation of pseudo labels for an unlabeled dataset and it can achieve promising detection results. Code has been made available at: https://github.com/MeriemSebai/MaskMitosis.
Similar content being viewed by others
References
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/. (Software available from tensorflow.org)
Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. https://github.com/matterport/Mask_RCNN
Ben-Cohen A, Diamant I, Klang E, Amitai M, Greenspan H (2016) Fully convolutional network for liver segmentation and lesions detection. In: Deep learning and data labeling for medical applications, pp 77–85. Springer
Biswas M, Kuppili V, Saba L, Edla D R, Suri H S, Sharma A, Cuadrado-Godia E, Laird J R, Nicolaides A, Suri J S (2019) Deep learning fully convolution network for lumen characterization in diabetic patients using carotid ultrasound: a tool for stroke risk. Med Biol Eng Comput 57(2):543–564
Chen H, Dou Q, Wang X, Qin J, Heng P A, et al. (2016) Mitosis detection in breast cancer histology images via deep cascaded networks. In: AAAI, pp 1160–1166
Chen H, Wang X, Heng P A (2016) Automated mitosis detection with deep regression networks. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp 1204–1207. IEEE
Chollet F et al (2015) Keras . https://github.com/fchollet/keras
Cireṡan DC, Giusti A, Gambardella L M, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 411–418. Springer
Cui S, Mao L, Jiang J, Liu C, Xiong S (2018) Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. J Healthc Eng 2018:4940593
Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. In: Annual Conference on Medical Image Understanding and Analysis, pp 506–517. Springer
Elston C W, Ellis I O (1991) Pathological prognostic factors in breast cancer. I. the value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology 19(5):403–410
Fu H, Cheng J, Xu Y, Zhang C, Wong D W K, Liu J, Cao X (2018) Disc-aware ensemble network for glaucoma screening from fundus image. IEEE Trans Med Imaging 37(11):2493–2501
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(1):2096–2030
García-Zapirain B, Elmogy M, El-Baz A, Elmaghraby AS (2018) Classification of pressure ulcer tissues with 3D convolutional neural network. Med Biol Eng Comput 56(12):2245–2258
Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 580–587
Giusti A, Caccia C, Cireṡari DC, Schmidhuber J, Gambardella L M (2014) A comparison of algorithms and humans for mitosis detection. In: 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), pp 1360–1363. IEEE
Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) CE-Net: context encoder network for 2D medical image segmentation. arXiv:1903.02740
He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Computer Vision (ICCV), 2017 IEEE International Conference on, pp 2980–2988. IEEE
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778
Huang C H, Lee H K (2012) Automated mitosis detection based on exclusive independent component analysis. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp 1856–1859. IEEE
Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7014–7023
Irshad H (2013) Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J Pathol Inform 4:10
Jader G, Fontineli J, Ruiz M, Abdalla K, Pithon M, Oliveira L (2018) Deep instance segmentation of teeth in panoramic x-ray images. In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp 400–407. IEEE
Johnson J W (2018) Adapting Mask-RCNN for automatic nucleus segmentation. arXiv:1805.00500
Khan A M, El-Daly H, Rajpoot N M (2012) A gamma-Gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp 149–152. IEEE
Krizhevsky A, Sutskever I, Hinton G E (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Lafarge M, Pluim J, Eppenhof K, Veta M (2019) Learning domain-invariant representations of histological images. Front Med 6:162
Lafarge M W, Pluim J P, Eppenhof K A, Moeskops P, Veta M (2017) Domain-adversarial neural networks to address the appearance variability of histopathology images. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 83–91. Springer
Li C, Wang X, Liu W (2017) Neural features for pedestrian detection. Neurocomputing 238:420–432
Li C, Wang X, Liu W, Latecki L J (2018) Deepmitosis: mitosis detection via deep detection, verification and segmentation networks. Med Image Anal 45:121–133
Lin T Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: CVPR, vol 1, p. 4
Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp 740–755. Springer
Liu J, Li P (2018) A mask R-CNN model with improved region proposal network for medical ultrasound image. In: International Conference on Intelligent Computing, pp 26–33. Springer
Liu M, Dong J, Dong X, Yu H, Qi L (2018) Segmentation of lung nodule in CT images based on mask R-CNN. In: 2018 9th International Conference on Awareness Science and Technology (iCAST), pp 1–6. IEEE
Liu X, Jiang D, Wang M, Song Z (2018) Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks. Med Biol Eng Comput 57:1–12
Liu Y, Zhang P, Song Q, Li A, Zhang P, Gui Z (2018) Automatic segmentation of cervical nuclei based on deep learning and a conditional random field. IEEE Access 6:53709–53721
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440
Macenko M, Niethammer M, Marron J S, Borland D, Woosley J T, Guan X, Schmitt C, Thomas N E (2009) A method for normalizing histology slides for quantitative analysis. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009. ISBI’09, pp 1107–1110. IEEE
Malon C D, Cosatto E (2013) Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inf 4:9
MITOS-ATYPIA-14: Mitos-atypia-14-dataset https://mitos-atypia-14.grand-challenge.org/dataset/ https://mitos-atypia-14.grand-challenge.org/dataset/ (2014). (Online; accessed 19.02.04)
Otálora S, Atzori M, Andrearczyk V, Khan A, Müller H. (2019) Staining invariant features for improving generalization of deep convolutional neural networks in computational pathology. Front Bioeng Biotech 7:198
Pang S, Du A, Orgun M A, Yu Z (2019) A novel fused convolutional neural network for biomedical image classification. Med Biol Eng Comput 57(1):107–121
Paul A, Dey A, Mukherjee D P, Sivaswamy J, Tourani V (2015) Regenerative random forest with automatic feature selection to detect mitosis in histopathological breast cancer images. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 94–102. Springer
Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 234–241. Springer
Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Naour G L, Gurcan M N (2013) Mitosis detection in breast cancer histological images an ICPR 2012 contest. J Pathol Inform 4:8
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg A C, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. 10.1007/s11263-015-0816-y
Sommer C, Fiaschi L, Hamprecht F A, Gerlich D W (2012) Learning-based mitotic cell detection in histopathological images. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp 2306–2309. IEEE
Tang P, Wang X, Huang Z, Bai X, Liu W (2017) Deep patch learning for weakly supervised object classification and discovery. Pattern Recogn 71:446–459
Tashk A, Helfroush M S, Danyali H, Akbarzadeh M (2013) An automatic mitosis detection method for breast cancer histopathology slide images based on objective and pixel-wise textural features classification. In: 2013 5th Conference on Information and Knowledge Technology (IKT), pp 406–410. IEEE
Tek F B (2013) Mitosis detection using generic features and an ensemble of cascade AdaBoosts. J Pathol Inform 4:12
TUPAC16: Tumor-proliferation-assessment-challenge. http://tupac.tue-image.nl/ http://tupac.tue-image.nl/ (2016). (Online; accessed 19.02.04)
Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171
Veta M, van Diest P J, Pluim J P (2013) Detecting mitotic figures in breast cancer histopathology images. In: Medical imaging 2013: digital pathology, vol 8676, p. 867607. International Society for Optics and Photonics
Veta M, Van Diest P J, Willems S M, Wang H, Madabhushi A, Cruz-Roa A, Gonzalez F, Larsen A B, Vestergaard J S, Dahl A B et al (2015) Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 20(1):237–248
Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, Tomaszewski J, Gonzalez F, Madabhushi A (2014) Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection. In: Medical imaging 2014: digital pathology, vol 9041, p. 90410B. International society for optics and photonics
Wang J, Wang X, Liu W (2018) Weakly-and semi-supervised faster R-CNN with curriculum learning. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp 2416–2421. IEEE
Wang X, Yan Y, Tang P, Bai X, Liu W (2018) Revisiting multiple instance neural networks. Pattern Recogn 74:15–24
Zhao H, Qi X, Shen X, Shi J, Jia J (2018) ICNet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 405–420
Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: CVPR
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sebai, M., Wang, X. & Wang, T. MaskMitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images. Med Biol Eng Comput 58, 1603–1623 (2020). https://doi.org/10.1007/s11517-020-02175-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11517-020-02175-z