Skip to main content
SpringerLink
Log in

Multi-scale skip-connection network for image super-resolution

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

A skip-connection learning framework-based convolution neural network (CNN) has recently achieved great success in image super-resolution (SR). However, most CNN models based on the skip-connection learning framework do not fully make use of potential multi-scale features of images. In this paper, we propose a multi-scale skip-connection network (MSN) to improve the visual quality of the image SR. First, convolution kernels with different sizes are exploited to capture the multi-scale features of LR images. All the feature-maps captured by convolution kernels of the same size are direct input into a multi-scale hybrid group (MHG); second, the convolution layers of each MHG are composed of dilated convolutions and standard convolutions. The hybrid convolutions can fully train feature details obtained from preceding and current scale convolution layers; three, the output of each hybrid convolution layer is fed into subsequent hybrid convolution layers by skip-connections, thus producing dense connections; lastly, the meta-upscale module is used as the upscale module, which can magnify the trained feature maps arbitrary scale factors. By being evaluated on a wide variety of images, the proposed MSN network achieves an advantage over the state-of-the-art methods in terms of both numerical results and visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Jia, B., Lv, J., Liu, D.: Deep learning-based automatic downbeat tracking: a brief review. Multimedia Syst. 25(6), 617–638 (2019)

    Article  Google Scholar 

  2. Yan, C., Gong, B., Wei, Y., Gao, Y.: Deep multi-view enhancement hashing for image retrieval. IEEE Trans Pattern Analy Machine Intelligence 14(8), 1–8 (2020)

    Google Scholar 

  3. Li, Q., Li, Z., Jeon, G.W.G., Yang, X. M.: Gated multiple feedback network for image super-resolution. In BMVC (2019)

  4. Zhang, L., Wu, X.: An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process 15(8), 2226–2238 (2006)

    Article  Google Scholar 

  5. Zhang, X., Wu, X.: Image interpolation by adaptive 2D autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process. 17(6), 887–896 (2008)

    Article  MathSciNet  Google Scholar 

  6. Zhang, K., Gao, X., Tao, D., Li, X.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21(11), 4544–4556 (2012)

    Article  MathSciNet  Google Scholar 

  7. Li, X., He, H., Wang, R., Tao, D.: Single image super-resolution via directional group sparsity and directional features. IEEE Trans. Image Process 24(1), 327–340 (2015)

    MATH  Google Scholar 

  8. Song, M., Chen, C., Bu, J., Sha, T.: Image-based facial sketch-to-photo synthesis via online coupled dictionary learning. Inf. Sci. 193(6), 233–246 (2012)

    Article  Google Scholar 

  9. Tao, D., Lin, X., Jin, L.: Principal component 2-D long short-term memory for font recognition on single Chinese characters. IEEE Trans. Cybern 46(3), 756–765 (2016)

    Article  Google Scholar 

  10. Gao, X., Zhang, K., Tao, D., Li, X.: Image super-resolution with sparse neighbor embedding. IEEE Trans. Image Process. 21(7), 3194–3205 (2012)

    Article  MathSciNet  Google Scholar 

  11. Dong, C., Loy, C.C., He, K., Tang X.: Learning a deep convolutional network for image super-resolution. In ECCV (2014)

  12. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans Pattern Analy Mach Intell 38(2), 295–307 (2016)

    Article  Google Scholar 

  13. Dong, C., Loy, C.C., Tang X.: Accelerating the super-resolution convolutional neural network. In ECCV (2016)

  14. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In CVPR (2016)

  15. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In CVPR (2016)

  16. Han, S., Mao, H., Dally, W, J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR (2016) 2

  17. Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M.: Enhanced deep residual networks for single image super-resolution. In CVPRW (2017)

  18. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In CVPR (2017)

  19. Huang, G., Liu, Z., Weinberger K. Q., van der Maaten and L.: Densely connected convolutional networks. In CVPR (2017)

  20. Zhang, Y., Tian Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In CVPR (2018)

  21. Hu, X.C., Mu, H.Y., Zhang, X.Y., Wang, Z.L., Sun, J., Tan, T.N.: Meta-SR: A magnification-arbitrary network for super-resolution. In CVPR (2019)

  22. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, D., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR (2016)

  23. Tong, T., Li, G., Liu, X., & Gao, Q. Image super-resolution using dense skip connections. In ICCV (2017)

  24. Zhang, Z.D., Wang, X.R., Jung, C.K.: DCSR: Dilated convolutions for single image super-resolution. IEEE Trans. Image Process. 28(4), 1625–1635 (2019)

    Article  MathSciNet  Google Scholar 

  25. Divakar, N., Babu, R. V.: Image denoising via CNNs: An Adversarial Approach. In CVPRW (2017)

  26. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z.Rethinking the inception architecture for computer vision. In CVPR (2016)

  27. Li, X., Li, F., Fern, X., & Raich, R. Filter shaping for convolutional neural networks. In ICLR (2017)

  28. Yu, F., & Koltun, V.: Multi-scale context aggregation by dilated convolutions. In ICLR (2016)

  29. Ioffe, S., & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML (2015)

  30. Glorot, X., Bordes, A., and Bengio, Y.: Deep sparse rectifier neural networks. In AISTATS (2011)

  31. He, K., Zhang, Ren, S., Sun, J.: Deep residual learning for image recognition. In CVPR (2016)

  32. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  33. Sheikh, H.R., Bovik, A.C., de Veciana, G.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)

    Article  Google Scholar 

  34. Deng, J., Dong, W., Socher, R., Li, L. J., Li, F. F.: Imagenet: A large-scale hierarchical image database. In CVPR (2009)

  35. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H.: et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW (2017)

  36. Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi- Morel M. L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC (2012)

  37. Zeyde, R., Elad, M., and Protter, M.: On single image scale-up using sparse-representations. In Proc. 7th Int. Conf. Curves Surf, (2010)

  38. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV (2001)

  39. Huang, J.B., Singh, A., and Ahuja N.: Single image super-resolution from transformed self-exemplars. In CVPR, (2015)

  40. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia tools and applications 76(20), 21811–21838 (2017)

    Article  Google Scholar 

  41. Kingma, D., and Ba Adam, J.: A method for stochastic optimization. In ICLR (2014)

  42. Yan, C., Shao, B., Zhao, H., Ning, R., Zhang, Y., Xu, F.: 3D room layout estimation from a single RGB image. IEEE Trans. Multimedia (2020). https://doi.org/10.1109/TMM.2020.2967645

    Article  Google Scholar 

  43. Yan, C., Li, Z., Zhang, Y., Liu, Y., Ji, X., and Zhang, Y.: Depth image denoising using nuclear norm and learning graph model. In arXiv:2008.03741, (2020), [online] Available: https://arxiv.org/abs/2008.03741

  44. Li, Z., Li, Q., Wu, W., Yang, J.L., Yang, X.M.: Deep recursive up-down sampling networks for single image super-resolution. Neurocomputing 398(20), 377–388 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China under Grant 61472319, and by Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2019JM-467.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Xi’an University of Technology, Xi’an, China

    Jing Liu, Jianhui Ge, Yuxin Xue, Wenjuan He, Qindong Sun & Shancang Li

  2. Department of Computer Science and Creative Technologies, University of the West of England, Bristol, UK

    Shancang Li

Authors
  1. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianhui Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuxin Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenjuan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qindong Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shancang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Ge, J., Xue, Y. et al. Multi-scale skip-connection network for image super-resolution. Multimedia Systems 27, 821–836 (2021). https://doi.org/10.1007/s00530-020-00712-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-020-00712-2

Keywords

Search

Navigation