Skip to main content
SpringerLink
Log in

\(\hbox {S}^2\hbox {RGAN}\): sonar-image super-resolution based on generative adversarial network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

As an important display mode of underwater environments, the sonar image has limitations on the resolution, which often leads to problems with low resolution of underwater objects. Therefore, the image super-resolution algorithm is needed to transform the images from low-resolution to high-resolution. It can improve the visual effect and contribute to subsequent processing such as 3D reconstruction and object recognition. This paper proposes a method for sonar image super-resolution based on generative adversarial networks (GAN). By comparing the super-resolution effects of various interpolation and convolutional neural network algorithms on sonar images, a Residual-in-Residual Dense Block network is employed as the generator of GAN since it has the low distortion and high perceptual quality. Because the sonar image training set does not have enough data, the generator utilizes the transfer learning on the sonar images to produce an optimized network model which is more suitable for super-resolution of sonar image. The VGG19 network is employed as the discriminator. In addition, the perceptual loss is introduced into the loss function of \(\hbox {S}^2\hbox {RGAN}\) to further improve the perceptual quality of super-resolution images. The experimental results indicate that the proposed \(\hbox {S}^2\hbox {RGAN}\) shows excellent performance. The generated super-resolution images of \(\hbox {S}^2\hbox {RGAN}\) have the remarkable advantages of both lower distortion and higher perceptual quality comparing with other methods. Because \(\hbox {S}^2\hbox {RGAN}\) focuses more on the reality and overall visual effect of super-resolution sonar images, it is suitable for various underwater situations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Zhang, X., Tang, M., Tong, R.: Robust super resolution of compressed video. Vis. Comput. 28(12), 1167–1180 (2012)

    Article  Google Scholar 

  2. Pandey, G., Ghanekar, U.: Classification of priors and regularization techniques appurtenant to single image super-resolution. Vis. Comput. 36(6), 1291–1304 (2019). https://doi.org/10.1007/s00371-019-01729-z

    Article  Google Scholar 

  3. XuXin, K., Yang, W.X., He, S., Zhang, Q., Yin, B., Wei, X., Lau, R.W.H.: Efficient image super-resolution integration. Vis. Comput. 34(6–8), 1065–1076 (2018)

    Google Scholar 

  4. Mikaeli, E., Aghagolzadeh, A., Azghani, M.: Single-image super-resolution via patch-based and group-based local smoothness modeling. Vis. Comput. 1–17 (2019). https://doi.org/10.1007/s00371-019-01756-w

  5. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 29(6), 1153–1160 (1981)

    Article  MathSciNet  Google Scholar 

  6. Stark, H., Oskoui, P.: High resolution image recovery from low resolution detector arrays. In: Digital Image Synthesis and Inverse Optics, pp. 80–84 (2010)

  7. Yu, J., Xiao, C., Su, K.: A method of gibbs artifact reduction for POCS super-resolution image reconstruction. In: International Conference on Signal Processing, pp. 1–4 (2006)

  8. Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP Graph. Models Image Process. 53(3), 231–239 (1991)

    Article  Google Scholar 

  9. Dai, S., Han, M., Wu, Y., et al.: Bilateral back-projection for single image super resolution. In: IEEE International Conference on Multimedia and Expo, pp. 1039–1042 (2007)

  10. Rasti, P., Demirel, H., Anbarjafari, G.: Improved iterative back projection for video super-resolution. In: Signal Processing and Communications Applications Conference, pp. 552–555 (2014)

  11. Park, J., Ku, B., Jin, Y., et al.: Side scan sonar image super resolution via region-selective sparse coding. IEICE Trans. Inf. Syst. 102(1), 210–213 (2019)

    Article  Google Scholar 

  12. Hurtós, N., Petillot, Y., Salvi, J.: Fourier-based registrations for two-dimensional forward-looking sonar image mosaicing. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5298–5305 (2012)

  13. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: Annual Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)

  14. Dong, C., Loy, C.C., He, K., et al.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  15. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407 (2016)

  16. Shi, W., Caballero, J., Husz, F., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  17. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  18. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  19. Lim, B., Son, S., Kim, H., et al.: Enhanced deep residual networks for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140 (2017)

  20. Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., Schroers, C.: A fully progressive approach to single-image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 977–986 (2018)

  21. Mirza, M., Osindero, S.: Conditional generative adversarial nets. In: Computer Science, pp. 2672–2680 (2014)

  22. Isola, P., Zhu, J.Y., Zhou, T., Alexei, A., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)

  23. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114 (2017)

  24. Wang, X., Ke, Y., Wu, S., Gu, J., Liu, Y., Dong, C., Change Loy, C., Qiao, Y., Tang, X.: ESRGAN: enhanced super-resolution generative adversarial networks. In: European Conference on Computer Vision, pp. 63–79 (2019)

  25. Sung, M., Joe, H., Kim, J., Yu, S.-C.: Convolutional neural network based resolution enhancement of underwater sonar image without losing working range of sonar sensors. IN: OCEANS-MTS/IEEE Kobe Techno-Oceans, pp. 1–6 (2018)

  26. Sung, M., Kim, J., Yu, S.C.: Image-based super resolution of underwater sonar images using generative adversarial network. In: TENCON 2018 IEEE Region 10 Conference, pp. 0457-0461 (2018)

  27. Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)

  28. Blau, Y., Mechrez, R., Timofte, R., et al.: The 2018 PIRM challenge on perceptual image super-resolution. In: European Conference on Computer Vision, pp. 334–355 (2019)

  29. Huang, G., Liu, Z., Van Der Maaten, L., et al.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  30. Shi, W., Caballero, J., Huszár, F., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. IN: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  31. Timofte, R., Agustsson, E., Van Gool, L., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

  32. Deng, J., Dong, W., Socher, R., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

  33. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556

  34. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)

    Article  Google Scholar 

  35. Ma, C., Yang, C.-Y., Yang, X., et al.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017)

    Article  Google Scholar 

  36. Kim, J., Kwon Lee, J., Mu, Lee K.: Deeply-recursive convolutional network for image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

  37. Isola, P., Zhu, J.Y., Zhou, T., et al.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  38. https://pan.baidu.com/s/10BOafbgHVXrbVqtVUP3Dvw Password: 6xs5 (2020)

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China (No. 61501132), China Postdoctoral Science Foundation (No. 2019M661319), Heilong-jiang Postdoctoral Scientific Research Developmental Founda-tion (No. LBH-Q17042), and Fundamental Research Funds for the Central Universities (No. 3072019CFT0603).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Harbin Engineering University, Harbin, 150001, China

    Hongtao Song, Minghao Wang, Liguo Zhang, Yang Li, Zhengyi Jiang & Guisheng Yin

Authors
  1. Hongtao Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minghao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liguo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhengyi Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guisheng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liguo Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, H., Wang, M., Zhang, L. et al. \(\hbox {S}^2\hbox {RGAN}\): sonar-image super-resolution based on generative adversarial network. Vis Comput 37, 2285–2299 (2021). https://doi.org/10.1007/s00371-020-01986-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01986-3

Keywords

Search

Navigation