Skip to main content
SpringerLink
Log in

A weighted feature transfer gan for medical image synthesis

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Recent studies have shown that CycleGAN is a highly influential medical image synthesis model. However, the lack of sufficient constraints and the bottleneck layer in auto-encoder network usually lead to blurry image and meaningless features, which may affect medical judgment. In order to synthesize accurate and meaningful medical images, weighted feature transfer GAN (WFT-GAN) is proposed to improve the quality of generated medical image, which is applied to the synthesis of unpaired multi-modal data. WFT-GAN adopts weighted feature transfer (WFT) instead of traditional skip connection to reduce the interference of encoding information on image decoding, while retaining the advantage of skip connection to the information transmission of the generated image. Moreover, the local perceptual adversarial loss combines the VGG feature map and adversarial model to make the local features of the image more meaningful. Experiments in three data sets show that the method in this paper can synthesize higher-quality medical images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Wang, S.H., et al.: Alcoholism identification based on an alexnet transfer learning model. Front. Psychiatry (2019). https://doi.org/10.3389/fpsyt.2019.00205

    Article  Google Scholar 

  2. Wang, S.-H., Muhammad, K., Hong, J., Sangaiah, A.K., Zhang, Y.-D.: Alcoholism identification via convolutional neural network based on parametric ReLU, dropout, and batch normalization. Neural Comput. Appl. 32(3), 665–680 (2020). https://doi.org/10.1007/s00521-018-3924-0

    Article  Google Scholar 

  3. Wang, S., Sun, J., Mehmood, I., Pan, C., Chen, Y., Zhang, Y.-D.: Cerebral micro-bleeding identification based on a nine-layer convolutional neural network with stochastic pooling. Concurr. Comput. Pract. Exp. 32(1), e5130 (2020). https://doi.org/10.1002/cpe.5130

    Article  Google Scholar 

  4. Wang, S.-H., Zhang, Y.-D., Yang, M., Liu, B., Ramirez, J., Gorriz, J.M.: Unilateral sensorineural hearing loss identification based on double-density dual-tree complex wavelet transform and multinomial logistic regression. Integr. Comput.-Aided Eng. 26(4), 411–426 (2019). https://doi.org/10.3233/ICA-190605

    Article  Google Scholar 

  5. Wu, H., Jiang, X., Jia, F.: UC-GAN for MR to CT image synthesis. In: Artificial Intelligence in Radiation Therapy, pp. 146–153. Springer, Cham (2019)

    Chapter  Google Scholar 

  6. Zhao, H., Li, H., Maurer-Stroh, S., Guo, Y., Deng, Q., Cheng, L.: Supervised segmentation of un-annotated retinal fundus images by synthesis. IEEE Trans. Med. Imaging 38(1), 46–56 (2019). https://doi.org/10.1109/TMI.2018.2854886

    Article  Google Scholar 

  7. Jin, C.-B., et al.: Deep CT to MR synthesis using paired and unpaired data. Sensors (2019). https://doi.org/10.3390/s19102361

    Article  Google Scholar 

  8. Xiang, L., et al.: Deep embedding convolutional neural network for synthesizing CT image from T1-Weighted MR image. Med. Image Anal. 47, 31–44 (2018). https://doi.org/10.1016/j.media.2018.03.011

    Article  Google Scholar 

  9. J.-Y. Zhu, Park, T., Isola, P., Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE international conference on computer vision (ICCV), Oct pp. 2242–2251. (2017) Doi: https://doi.org/10.1109/ICCV.2017.244

  10. Fabius, O., van Amersfoort J.R. Variational Recurrent Auto-Encoders. ArXiv14126581 Cs Stat, (2015), Accessed May 10, 2020. [Online]. Available: https://arxiv.org/abs/1412.6581

  11. Cai, X., Li, S., Liu, X., Han, G.: Vision-based fall detection with multi-task hourglass convolutional auto-encoder. IEEE Access 8, 44493–44502 (2020). https://doi.org/10.1109/ACCESS.2020.2978249

    Article  Google Scholar 

  12. Choi, Y., Choi, M. Kim, M., J.-W. Ha, Kim, S., Choo, J. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, pp. 8789–8797 (2018). doi: https://doi.org/10.1109/CVPR.2018.00916

  13. Liu, M., et al., STGAN: a unified selective transfer network for arbitrary image attribute editing. pp. 3673–3682 (2019) Accessed: Aug. 18, 2020. [Online]. Available: https://openaccess.thecvf.com/content_CVPR_2019/html/Liu_STGAN_A_Unified_Selective_Transfer_Network_for_Arbitrary_Image_Attribute_CVPR_2019_paper.html.

  14. Liu, M., et al., “STGAN: a unified selective transfer network for arbitrary image attribute editing. pp. 3673–3682 ( 2019) Accessed: Jul. 15, 2020. [Online]. Available: https://openaccess.thecvf.com/content_CVPR_2019/html/Liu_STGAN_A_Unified_Selective_Transfer_Network_for_Arbitrary_Image_Attribute_CVPR_2019_paper.html

  15. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A. Self-attention generative adversarial networks. In international conference on machine learning. May (2019), pp. 7354–7363, Accessed: Jul. 15, 2020. [Online]. Available: https://proceedings.mlr.press/v97/zhang19d.html

  16. Mechrez, R., Talmi, I., Zelnik-Manor, L. The contextual loss for image transformation with non-aligned data. In: Computer Vision–ECCV 2018, Cham, pp. 800–815 (2018). doi: 10.1007/978-3-030-01264-9_47

  17. Zhang, X., Ng, R., Chen, Q. Single image reflection separation with perceptual losses. In 2018 IEEE/CVF conference on computer vision and pattern recognition pp. 4786–4794 (2018). doi: https://doi.org/10.1109/CVPR.2018.00503

  18. Wang, L., Guo, S., Huang, W., Qiao, Y. Places205-VGGNet models for scene recognition. ArXiv150801667 Cs, Aug. (2015), Accessed: Aug. 17, 2020. [Online]. Available: https://arxiv.org/abs/1508.01667

  19. Joyce, T., Chartsias, A., Tsaftaris, S.A. Robust multi-modal MR image synthesis. In: Medical Image Computing and Computer Assisted Intervention−MICCAI 2017, Cham: Springer pp. 347–355 (2017) 10.1007/978-3-319-66179-7_40

  20. Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., Greenspan, H. Synthetic data augmentation using GAN for improved liver lesion classification. In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), Apr 2018, pp. 289–293, https://doi.org/10.1109/ISBI.2018.8363576

  21. Ouerghi, H., Mourali, O., Zagrouba, E.: Non-subsampled shearlet transform based MRI and PET brain image fusion using simplified pulse coupled neural network and weight local features in YIQ colour space. IET Image Process. 12(10), 1873–1880 (2018). https://doi.org/10.1049/iet-ipr.2017.1298

    Article  Google Scholar 

  22. Costa, P., Galdran, A., Meyer, M.I., Mendonça, A.M., Campilho, A. Adversarial synthesis of retinal images from vessel trees. In Image Analysis and Recognition. Cham: Springer pp. 516–523 (2017). https://doi.org/10.1007/978-3-319-59876-5_57

  23. Lample, G., Zeghidour, N., Usunier, N., Bordes, A., DENOYER,L., Ranzato, M. A., Fader Networks:Manipulating Images by Sliding Attributes. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, (Eds.) Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pp. 5967–5976 (2017)

  24. Zhou, S., Nie, D., Adeli, E., Yin, J., Lian, J., Shen, D.: High-resolution encoder–decoder networks for low-contrast medical image segmentation. IEEE Trans. Image Process. 29, 461–475 (2020). https://doi.org/10.1109/TIP.2019.2919937

    Article  MathSciNet  Google Scholar 

  25. Ronneberger, O., Fischer, P., Brox, T. U-Net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Cham, pp. 234–241 (2015) doi: 10.1007/978-3-319-24574-4_28

  26. Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury,S., Pal, C.The importance of skip connections in biomedical image segmentation. In Deep Learning and Data Labeling for Medical Applications, Cham pp. 179–187 (2016) doi: https://doi.org/10.1007/978-3-319-46976-8_19

  27. Mechrez, R., Talmi, I., Zelnik-Manor, L., The contextual loss for image transformation with non-aligned data. ArXiv180302077 Cs, Jul. (2018). Accessed: May 20, 2020. [Online]. Available: https://arxiv.org/abs/1803.02077

  28. Mechrez, R., Talmi, I.,Shama, F., Zelnik-Manor, L. Maintaining natural image statistics with the contextual loss. In Computer Vision–ACCV 2018, Dec, pp. 427–443 (2018) doi: https://doi.org/10.1007/978-3-030-20893-6_27

  29. Yang, Q., H.-X. Yu, Wu, A., W.-S. Zheng. Patch-based discriminative feature learning for unsupervised person re-identification. pp. 3633–3642 (2019) Accessed: Sep. 23, 2020. [Online]. Available: https://openaccess.thecvf.com/content_CVPR_2019/html/Yang_Patch-Based_Discriminative_Feature_Learning_for_Unsupervised_Person_Re-Identification_CVPR_2019_paper.html

  30. Goodfellow, I.J. et al. Generative adversarial networks ArXiv14062661 Cs Stat, Jun (2014), Accessed: May 10, 2020. [Online]. Available: https://arxiv.org/abs/1406.2661

  31. Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J. Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Machine Learning. Jul (2017), pp. 1857–1865, Accessed: May 11, 2020. [Online]. Available: https://proceedings.mlr.press/v70/kim17a.html

  32. Kingma, D.P., Ba, J. Adam: a method for stochastic optimization. ArXiv14126980 Cs, Jan (2017), Accessed: May 11, 2020. [Online]. Available: https://arxiv.org/abs/1412.6980

  33. Kavur, A.E., et al. CHAOS Challenge–combined (ct-mr) healthy abdominal organ segmentation. ArXiv200106535 Cs Eess, May 2020, Accessed: Aug. 18, 2020. [Online]. Available: https://arxiv.org/abs/2001.06535

  34. “DRIVE-Grand Challenge,” grand-challenge.org. https://drive.grand-challenge.org/(accessed Aug. 18, 2020)

  35. Chen, J., Tian, J., Lee, N., Zheng, J., Smith, R.T., Laine, A.F.: A partial intensity invariant feature descriptor for multimodal retinal image registration. IEEE Trans. Biomed. Eng. 57(7), 1707–1718 (2010). https://doi.org/10.1109/TBME.2010.2042169

    Article  Google Scholar 

  36. Hernandez-Matas, C., Zabulis, X., Argyros, A.A. Retinal image registration through simultaneous camera pose and eye shape estimation. In 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Orlando, FL, USA, Aug. 2016, pp. 3247–3251, doi: https://doi.org/10.1109/EMBC.2016.7591421

Download references

Acknowledgements

The work is partially supported by the Natural Science Foundation of China (Nos. 61503188), CERNET Innovation Project (NGII20180604) and the Natural Science Foundation of Jiangsu Province (Nos. BK20180727).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Nanjing Normal University, Nanjing, Jiangsu Province, China

    Shuaizhen Yao, Yi Chen & Yanhui Gu

  2. School of Public Administration, Nanjing Agricultural University, Nanjing, Jiangsu Province, China

    Jianhua Tan

  3. Nanjing Vocational University of Industry Technology, Nanjing, Jiangsu Province, China

    Jianhua Tan

Authors
  1. Shuaizhen Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianhua Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanhui Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yi Chen or Yanhui Gu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yao, S., Tan, J., Chen, Y. et al. A weighted feature transfer gan for medical image synthesis. Machine Vision and Applications 32, 22 (2021). https://doi.org/10.1007/s00138-020-01152-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01152-8

Keywords

Search

Navigation