Abstract
In this paper, we propose HybridGAN – a new medical MR image synthesis methods via generative adversarial learning. Specifically, our synthesizer generates MRI data in a sequential manner: first in order to improve the robustness of image synthesis, an input full-size real MR image is divided into an array of sub-images. Then, to avoid overfitting limited MRI encodings, these sub-images and an unlimited amount of random latent noise vectors become the input of automatic encoder for learning the marginal image distributions of real images. Finally, pseudo patches with constrained noise vectors are put into RU-NET which is a component of our HybridGAN to generate a large number of synthetic MR images. In RU-NET, A SpliceLayer is then employed to fuse sub-images together in an interlaced manner into a full-size image. The experimental results show that HybridGAN can effectively synthesize a large variety of MR images and display a good visual quality. Compared to the state-of-the-art synthesis methods, our method achieves a significant improvement in terms of both visual and quantitative evaluation metrics.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Benesty J, Chen J, Huang Y et al (2009) Pearson correlation coefficient[M]//noise reduction in speech processing. Springer, Berlin, pp 1–4
Bińkowski M, Sutherland DJ, Arbel M et al (2018) Demystifying mmd gans[J]. arXiv preprint arXiv:1801.01401
Bousquet O, Gelly S, Tolstikhin I et al (2017) From optimal transport to generative modeling: the VEGAN cookbook[J]. arXiv preprint arXiv:1705.07642
Danielsson PE (1980) Euclidean distance mapping[J]. Computer Graphics and image processing 14(3):227–248
Genevay A, Peyré G, Cuturi M (2017) GAN and VAE from an optimal transport point of view[J]. arXiv preprint arXiv:1706.01807
Goodfellow I, Pouget-Abadie J, Mirza M et al Generative adversarial nets[C]. Advances in neural information processing systems. 2014:2672–2680
Gulrajani I, Ahmed F, Arjovsky M et al Improved training of wasserstein gans[C]. Advances in neural information processing systems. 2017:5767–5777
Han C, Hayashi H, Rundo L et al GAN-based synthetic brain MR image generation[C]//2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). IEEE, 2018:734–738
Heusel M, Ramsauer H, Unterthiner T, Nessler B, Klambauer G, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a nash equilibrium. arXiv preprint arXiv:1706.08500
Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In CVPR
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization[J]. arXiv preprint arXiv:1412.6980
Kommrusch S, Pouchet LN (2018) Synthetic lung nodule 3d image generation using autoencoders[J]. arXiv preprint arXiv:1811.07999
Lau F, Hendriks T, Lieman-Sifry J et al (2018) Scargan: chained generative adversarial networks to simulate pathological tissue on cardiovascular MR scans[M]//deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 343–350
Li C, Wand M (2016) Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision, pages 702–716. Springer
Li Z, Zhang T, Zhang D (2019) SEGAN: Structure-Enhanced Generative Adversarial Network for Compressed Sensing MRI Reconstruction[J]. arXiv preprint arXiv:1902.06455
Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B (2015) Adversarial autoencoders. arXiv preprint arXiv:1511.05644
Mirza M, Osindero S (2014) Conditional generative adversarialnets. arXiv preprint arXiv:1411.1784
Odena A, Olah C, Shlens J (2017) Conditional image synthesis with auxiliary classifier GANs. In ICML
Olut S, Sahin YH, Demir U et al (2018) Generative adversarial training for MRA image synthesis using multi-contrast MRI[C]//international workshop on PRedictive intelligence in MEdicine. Springer, Cham, pp 147–154
Pan Y, Liu M, Lian C et al (2018) Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer’s disease diagnosis[C]//international conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 455–463
Pathak D, Krahenbuhl P, Donahue J et al. Context encoders: Feature learning by inpainting[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:2536–2544
Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv preprint arXiv:1511.06434
Regmi K, Borji A (2018) Cross-view image synthesis using conditional gans. In CVPR
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation[C]//international conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241
Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X (2016) Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp 2234–2242
Saltzer JH, Reed DP, Clark DD (1984) End-to-end arguments in system design[J]. Technology 100:0661
Shin HC, Tenenholtz NA, Rogers JK et al (2018) Medical image synthesis for data augmentation and anonymization using generative adversarial networks[C]//international workshop on simulation and synthesis in medical imaging. Springer, Cham, pp 1–11
Sohn K, Lee H, Yan X (2015) Learning structured output representation using deep conditional generative models. In NeurIPS
Sohn K, Lee H, Yan X Learning structured output representation using deep conditional generative models[C]. Advances in neural information processing systems. 2015:3483–3491
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2818–2826
Wang Z, Lin Y, Cheng KT et al (2018) Semi-supervised mp-MRI Data Synthesis with StitchLayer and Auxiliary Distance Maximization[J]. arXiv preprint arXiv:1812.06625
White T (2016) Sampling generative networks. arXiv preprint a improved techniques for training gans rXiv:1609.04468
Yang X, Liu C, Wang Z, Yang J, Le Min H, Wang L, Cheng KTT (2017) Co-trained convolutional neural networks for automated detection of prostate cancer in multi-parametric mri. Med Image Anal 42:212–227
Yang X, Wang Z, Liu C, Le HM, Chen J, Cheng K-TT, Wang L (2017) Joint detection and diagnosis of prostate cancer in multiparametric mri based on multimodal convolutional neural networks. In International Conference on Medical Image Computing and Computer Assisted Intervention. Springer, pp 426–434
Zhao H, Li H, Maurer-Stroh S, Cheng L (2018) Synthesizing retinal and neuronal images with generative adversarial nets[J]. Med Image Anal 49:14–26
Zhu J-Y, Zhang R, Pathak D, Darrell T, Efros AA, Wang O, Shechtman E (2017) Toward multimodal image to-image translation. In NIPS
Zhu J-Y, Zhang R, Pathak D, Darrell T, Efros AA, Wang O, Shechtman E (2017) Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems pp 465–476
Acknowledgements
Chen’s research was sponsored by Hubei Provincial Department of Education under a Career Development Award No. D20181705. Xiao Qin’s work is supported by the U.S. National Science Foundation under Grants IIS-1618669, CCF-0845257 (CAREER), CNS-0917137, and OCI-0753305.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chen, J., Luo, S., Xiong, M. et al. HybridGAN: hybrid generative adversarial networks for MR image synthesis. Multimed Tools Appl 79, 27615–27631 (2020). https://doi.org/10.1007/s11042-020-09387-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09387-3