Skip to main content
SpringerLink
Log in

A multi-scale generative adversarial network for real-world image denoising

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

With the rising popularity and wide applications of image processing technologies in recent years, various deep learning methods have been proposed for image denoising. However, since most of such methods focus mainly on synthetic noises, their denoising effects on the spatially variant real-world noises could be further improved with more sophisticated network and training schemes. In this paper, a multi-scale generative adversarial network (MSGAN) that employs a novel network architecture and a well-designed training scheme is proposed. Specifically, a cascade multi-scale module is proposed as a basic building block of MSGAN to make use of the multi-scale context and increase the network learning capacity first, and then, a spatial attention mechanism is applied onto MSGAN to refine the denoising results. Finally, a sophisticated training scheme, which combines the pixel-level loss with the adversarial loss, is designed to suppress the real-world noises while restore both high-frequency and low-frequency image details simultaneously. Extensive experiments are conducted with several typical datasets to verify the effectiveness of MSGAN. Results demonstrate that MSGAN is promising for real-world image denoising in terms of both quantitative metrics (PSNR, SSIM) and visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Gonzales, R.C., Woods, R.E.: Digital Image Processing. Prentice-Hall, Inc., Upper Saddle River (2002)

    Google Scholar 

  2. Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19(9), 2345 (2010)

    Article  MathSciNet  Google Scholar 

  3. Heide, F., Steinberger, M., Tsai, Y.T., Rouf, M., Pajak, D., Reddy, D., Gallo, O., Liu, J., Heidrich, W., Egiazarian, K., et al.: Flexisp: a flexible camera image processing framework. ACM Trans. Graph. 33(6), 1 (2014)

    Article  Google Scholar 

  4. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080 (2007)

    Article  MathSciNet  Google Scholar 

  5. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005 (2005)

  6. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014)

  7. Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2233 (2012)

    Article  Google Scholar 

  8. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4(2), 460 (2005)

    Article  MathSciNet  Google Scholar 

  9. Xu, J., Osher, S.: Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Trans. Image Process. 16, 534 (2007)

    Article  MathSciNet  Google Scholar 

  10. Weiss, Y., Freeman, W.T.: What makes a good model of natural images? In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)

  11. Roth, S., Black, M.J.: Fields of Experts. Int. J. Comput. Vis. 82(2), 205 (2009)

    Article  Google Scholar 

  12. Anwar, S., Porikli, F., Huynh, C.P.: Category-specific object image denoising. IEEE Trans. Image Process. 26, 5506–5518 (2017)

    Article  MathSciNet  Google Scholar 

  13. Yue, H., Sun, X., Yang, J., Wu, F.: CID: combined image denoising in spatial and frequency domains using web images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2933–2940 (2014)

  14. Luo, E., Chan, S.H., Nguyen, T.Q.: Adaptive image denoising by targeted databases. IEEE Trans. Image Process. 24(7), 2167 (2015)

    Article  MathSciNet  Google Scholar 

  15. Anwar, S., Barnes, N.: Real image denoising with feature attention. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

  16. Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1712–1722 (2019)

  17. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26, 3142–3155 (2017)

    Article  MathSciNet  Google Scholar 

  18. Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN based image denoising. IEEE Trans. Image Process. 27, 4608–4622 (2017)

    Article  MathSciNet  Google Scholar 

  19. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  20. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  21. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)

  22. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  23. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

  24. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML, vol. 30, p. 3 (2013)

  25. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation, In: International Conference on Medical Image Computing and Computer-Assisted Intervention pp. 234–241. Springer (2015)

  26. Liu, C., Shang, Z., Qin, A.: A multiscale image denoising algorithm based on dilated residual convolution network. In: Chinese Conference on Image and Graphics Technologies, pp. 193–203. Springer (2019)

  27. Bao, L., Yang, Z., Wang, S., Bai, D., Lee, J.: Real image denoising based on multi-scale residual dense block and cascaded U-Net with block-connection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)

  28. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)

  29. Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

  30. Gao, S., Cheng, M.M., Zhao, K., Zhang, X.Y., Torr, P.H.S.: Res2Net: a new multi-scale backbone architecture. IEEE Trans. Pattern Anal. Mach. Intell. 43, 652–662 (2021)

    Article  Google Scholar 

  31. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In:European Conference on Computer Vision (2018)

  32. Xu, J., Li, H., Liang, Z., Zhang, D., Zhang, L.: Real-world noisy image denoising: a new benchmark. arXiv preprint arXiv:1804.02603 (2018)

  33. Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In:Computer Vision & Pattern Recognition (2017)

  34. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 126–135 (2017)

  35. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  36. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)

  37. Ma, L., Ma, Y., Lin, Q., Ji, J., Coello, C.A.C., Gong, M.: SNEGAN: signed network embedding by using generative adversarial nets. IEEE Trans. Emerg. Top. Comput. Intell. (2020). https://doi.org/10.1109/TETCI.2020.3035937

    Article  Google Scholar 

  38. Cao, Y., Wu, X., Qi, S., Liu, X., Wu, Z., Zuo, W.: Pseudo-ISP: learning pseudo in-camera signal processing pipeline from a color image denoiser. arXiv preprint arXiv:2103.10234 (2021)

Download references

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, 710072, Shaanxi, People’s Republic of China

    Xiaojun Yu, Zixuan Fu & Chenkun Ge

Authors
  1. Xiaojun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zixuan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenkun Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaojun Yu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research is supported in part by the National Natural Science Foundation of China (61705184), the Key Research and Development Program of Shaanxi (2021SF-342), the Fundamental Research Funds for the Central Universities (G2018KY0308), China Postdoctoral Science Foundation (2018M641013), Postdoctoral Science Foundation of Shaanxi Province (2018BSHYDZZ05).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, X., Fu, Z. & Ge, C. A multi-scale generative adversarial network for real-world image denoising. SIViP 16, 257–264 (2022). https://doi.org/10.1007/s11760-021-01984-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01984-5

Keywords

Search

Navigation