Skip to main content
SpringerLink
Log in

Adversarial Confidence Learning for Medical Image Segmentation and Synthesis

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Generative adversarial networks (GAN) are widely used in medical image analysis tasks, such as medical image segmentation and synthesis. In these works, adversarial learning is directly applied to the original supervised segmentation (synthesis) networks. The usage of adversarial learning is effective in improving visual perception performance since adversarial learning works as realistic regularization for supervised generators. However, the quantitative performance often cannot improve as much as the qualitative performance, and it can even become worse in some cases. In this paper, we explore how we can take better advantage of adversarial learning in supervised segmentation (synthesis) models and propose an adversarial confidence learning framework to better model these problems. We analyze the roles of discriminator in the classic GANs and compare them with those in supervised adversarial systems. Based on this analysis, we propose adversarial confidence learning, i.e., besides the adversarial learning for emphasizing visual perception, we use the confidence information provided by the adversarial network to enhance the design of supervised segmentation (synthesis) network. In particular, we propose using a fully convolutional adversarial network for confidence learning to provide voxel-wise and region-wise confidence information for the segmentation (synthesis) network. With these settings, we propose a difficulty-aware attention mechanism to properly handle hard samples or regions by taking structural information into consideration so that we can better deal with the irregular distribution of medical data. Furthermore, we investigate the loss functions of various GANs and propose using the binary cross entropy loss to train the proposed adversarial system so that we can retain the unlimited modeling capacity of the discriminator. Experimental results on clinical and challenge datasets show that our proposed network can achieve state-of-the-art segmentation (synthesis) accuracy. Further analysis also indicates that adversarial confidence learning can both improve the visual perception performance and the quantitative performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://github.com/pytorch/pytorch.

  2. https://github.com/ginobilinie/medSynthesisV1.

  3. https://github.com/ginobilinie/medSegmentation.

  4. https://promise12.grand-challenge.org/.

  5. https://promise12.grand-challenge.org/evaluation/results/.

References

  • Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875.

  • Chaitanya, K., Karani, N., Baumgartner, C. F., Becker, A., Donati, O., & Konukoglu, E. (2019). Semi-supervised and task-driven data augmentation. In International conference on information processing in medical imaging (pp. 29–41). Springer.

  • Chen, C., Dou, Q., Chen, H., & Heng, P. A. (2018a). Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation. In International workshop on machine learning in medical imaging (pp. 143–151). Springer.

  • Chen, H., Qi, X., Yu, L., & Heng, P. A. (2016). DCAN: Deep contour-aware networks for accurate gland segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2487–2496).

  • Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018b). Encoder–decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).

  • Costa, P., Galdran, A., Meyer, M. I., Niemeijer, M., Abràmoff, M., Mendonça, A. M., et al. (2018). End-to-end adversarial retinal image synthesis. IEEE Transactions on Medical Imaging, 37(3), 781–791.

    Google Scholar 

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE TPAMI, 38(2), 295–307.

    Google Scholar 

  • Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In AISTATS (pp. 249–256).

  • Goodfellow, I., et al. (2014). Generative adversarial nets. In NIPS.

  • Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. C. (2017). Improved training of wasserstein GANs. In NIPS (pp. 5767–5777).

  • Guo, Y., et al. (2016). Deformable MR prostate segmentation via deep feature learning and sparse patch matching. IEEE TMI, 35, 1077–1089.

    Google Scholar 

  • Han, X. (2017). MR-based synthetic CT generation using a deep convolutional neural network method. Medical Physics, 44(4), 1408–1419.

    Google Scholar 

  • Hardy, C., Merrer, E. L., & Sericola, B. (2018). MD-GAN: Multi-discriminator generative adversarial networks for distributed datasets. arXiv preprint arXiv:1811.03850.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR (pp. 770–778).

  • Huang, Y., Shao, L., & Frangi, A. F. (2017). Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. arXiv preprint arXiv:1705.02596.

  • Hung, W. C., Tsai, Y. H., Liou, Y. T., Lin, Y. Y., & Yang, M. H. (2018). Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv:1802.07934.

  • Huo, Y., Xu, Z., Bao, S., Assad, A., Abramson, R. G., & Landman, B. A. (2018). Adversarial synthesis learning enables segmentation without target modality ground truth. In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018) (pp. 1217–1220). IEEE.

  • Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125–1134).

  • Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision (pp. 694–711). Springer.

  • Kohl, S., et al. (2017). Adversarial networks for the detection of aggressive prostate cancer. arXiv preprint arXiv:1702.08014.

  • Kumar, P., & Srivastava, M. M. (2018). Example mining for incremental learning in medical imaging. In 2018 IEEE symposium series on computational intelligence (SSCI) (pp. 48–51). IEEE.

  • Li, C., & Wand, M. (2016). Precomputed real-time texture synthesis with Markovian generative adversarial networks. In European conference on computer vision (pp. 702–716). Springer.

  • Lin, G., Milan, A., Shen, C., & Reid, I. (2017a). Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1925–1934).

  • Lin, T. Y., et al. (2017b). Focal loss for dense object detection. arXiv preprint arXiv:1708.02002.

  • Litjens, G., et al. (2014). Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge. MIA, 18(2), 359–373.

    Google Scholar 

  • Long, J., et al. (2015). Fully convolutional networks for semantic segmentation. In CVPR (pp. 3431–3440).

  • Luc, P., Couprie, C., Chintala, S., & Verbeek, J. (2016). Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408.

  • Lucic, M., Kurach, K., Michalski, M., Gelly, S., & Bousquet, O. (2018). Are GANs created equal? A large-scale study. In Advances in neural information processing systems (pp. 700–709).

  • Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Smolley, S. P. (2017). Least squares generative adversarial networks. In ICCV (pp. 2813–2821). IEEE.

  • Mathieu, M., Couprie, C., & LeCun, Y. (2015). Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440.

  • Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., et al. (2015). The multimodal brain tumor image segmentation benchmark (brats). IEEE TMI, 34(10), 1993.

    Google Scholar 

  • Merkow, J., Marsden, A., Kriegman, D., & Tu, Z. (2016). Dense volume-to-volume vascular boundary detection. In MICCAI (pp. 371–379). Springer.

  • Mescheder, L., Geiger, A., & Nowozin, S. (2018). Which training methods for GANs do actually converge? In ICML (pp. 3478–3487).

  • Metz, L., Poole, B., Pfau, D., & Sohl-Dickstein, J. (2016). Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163.

  • Milletari, F., et al. (2016). V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV (pp. 565–571). IEEE.

  • Moeskops, P., et al. (2017). Adversarial training and dilated convolutions for brain MRI segmentation. arXiv preprint arXiv:1707.03195.

  • Mordido, G., Yang, H., & Meinel, C. (2018). Dropout-GAN: Learning from a dynamic ensemble of discriminators. arXiv preprint arXiv:1807.11346.

  • Nie, D., Cao, X., Gao, Y., Wang, L., & Shen, D. (2016). Estimating CT image from MRI data using 3D fully convolutional networks. In DLMIA (pp. 170–178). Springer.

  • Nie, D., Trullo, R., Lian, J., Petitjean, C., Ruan, S., Wang, Q., & Shen, D. (2017). Medical image synthesis with context-aware generative adversarial networks. In MICCAI. (pp. 417–425). Springer.

  • Nie, D., Trullo, R., Lian, J., Wang, L., Petitjean, C., Ruan, S., et al. (2018a). Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12), 2720–2730.

    Google Scholar 

  • Nie, D., Wang, L., Gao, Y., Lian, J., & Shen, D. (2018b). Strainet: Spatially varying stochastic residual adversarial networks for MRI pelvic organ segmentation. IEEE Transactions on Neural Networks and Learning Systems, 30, 1552–1564.

    MathSciNet  Google Scholar 

  • Nie, D., Wang, L., Xiang, L., Zhou, S., Ehsan, A., & Shen, D. (2019). Difficulty-aware attention network with confidence learning for medical image segmentation. In AAAI.

  • Oktay, O., et al. (2018). Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999.

  • Pan, T., Wang, B., Ding, G., & Yong, J. H. (2017). Fully convolutional neural networks with full-scale-features for semantic segmentation.

  • Qi, G. J. (2017). Loss-sensitive generative adversarial networks on Lipschitz densities. arXiv preprint arXiv:1701.06264.

  • Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.

  • Ronneberger, O., et al. (2015). U-net: Convolutional networks for biomedical image segmentation. In MICCAI (pp. 234–241). Springer.

  • Roy, A. G., et al. (2015). Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks. In MICCAI.

  • Sabokrou, M., Pourreza, M., Fayyaz, M., Entezari, R., Fathy, M., Gall, J., & Adeli, E. (2018). Avid: Adversarial visual irregularity detection. In ACCV.

  • Shi, F., Cheng, J., Wang, L., Yap, P. T., & Shen, D. (2015). LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE Transactions on Medical Imaging, 34(12), 2459–2466.

    Google Scholar 

  • Shrivastava, A., Gupta, A., & Girshick, R. (2016). Training region-based object detectors with online hard example mining. In CVPR (pp. 761–769).

  • Sudre, C. H., et al. (2017). Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In DLMIA. Springer.

  • Tsai, Y. H., Hung, W. C., Schulter, S., Sohn, K., Yang, M. H., & Chandraker, M. (2018). Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7472–7481).

  • Vu, T. H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2517–2526).

  • Wolterink, J. M., et al. (2017). Generative adversarial networks for noise reduction in low-dose CT. TMI, 36(12), 2536–2545.

    Google Scholar 

  • Xiao, H., Wei, Y., Liu, Y., Zhang, M., & Feng, J. (2017). Transferable semi-supervised semantic segmentation. arXiv preprint arXiv:1711.06828.

  • Xue, Y., Xu, T., Zhang, H., Long, L. R., & Huang, X. (2018). Segan: Adversarial network with multi-scale \(L_1\) loss for medical image segmentation. Neuroinformatics, 16, 1–10.

    Google Scholar 

  • Xue, Y., Zhou, Q., Ye, J., Long, L. R., Antani, S., Cornwell, C., Xue, Z., & Huang, X. (2019). Synthetic augmentation and feature-based filtering for improved cervical histopathology image classification. In International conference on medical image computing and computer-assisted intervention (pp. 387–396). Springer.

  • Yang, Q., Yan, P., Zhang, Y., Yu, H., Shi, Y., Mou, X., et al. (2018). Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss. IEEE Transactions on Medical Imaging, 37(6), 1348–1357.

    Google Scholar 

  • Yang, X., Yu, L., Wu, L., Wang, Y., Ni, D., Qin, J., & Heng, P. A. (2017). Fine-grained recurrent neural networks for automatic prostate segmentation in ultrasound images. In AAAI (pp. 1633–1639).

  • Yu, F., Koltun, V., & Funkhouser, T. A. (2017a). Dilated residual networks.

  • Yu, L., et al. (2017b). Volumetric ConvNets with mixed residual connections for automated prostate segmentation from 3D MR images. In AAAI.

  • Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D. P., & Chen, D. Z. (2017). Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In MICCAI (pp. 408–416). Springer.

  • Zhang, Z., Yang, L., & Zheng, Y. (2018). Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 9242–9251).

  • Zhao, C., et al. (2018). A deep learning based anti-aliasing self super-resolution algorithm for MRI. In MICCAI (pp. 100–108). Springer.

  • Zhou, X. Y., Shen, M., Riga, C., Yang, G. Z., & Lee, S. L. (2017). Focal fcn: Towards small object segmentation with limited training data. arXiv preprint arXiv:1711.01506.

  • Zhu, Q., et al. (2019). Boundary-weighted domain adaptive neural network for prostate MR image segmentation. arXiv preprint arXiv:1902.08128.

  • Zhu, W., Xiang, X., Tran, T. D., Hager, G. D., & Xie, X. (2018). Adversarial deep structured nets for mass segmentation from mammograms. In ISBI (pp. 847–850). IEEE.

Download references

Acknowledgements

This work was supported by the National Institutes of Health under Grant R01 CA206100.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27514, USA

    Dong Nie

  2. Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27514, USA

    Dong Nie & Dinggang Shen

  3. Department of Brain and Cognitive Engineering, Korea University, Seoul, 02841, Republic of Korea

    Dinggang Shen

Authors
  1. Dong Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dinggang Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dong Nie or Dinggang Shen.

Additional information

Communicated by Jun-Yan Zhu, Hongsheng Li, Eli Shechtman, Ming-Yu Liu, Jan Kautz, Antonio Torralba.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nie, D., Shen, D. Adversarial Confidence Learning for Medical Image Segmentation and Synthesis. Int J Comput Vis 128, 2494–2513 (2020). https://doi.org/10.1007/s11263-020-01321-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01321-2

Keywords

Search

Navigation