Skip to main content
SpringerLink
Log in

A lightweight multi-scale feature integration network for real-time single image super-resolution

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Recently, numerous methods based on convolutional neural networks (CNNs) have been proposed to attain satisfactory performance in single image super-resolution (SISR). Meanwhile, diverse lightweight CNN-based networks have been proposed to achieve applicability in real-time scenarios. However, the receptive fields in lightweight networks are limited because they do not make good use of multi-scale information. In this paper, we propose a lightweight multi-scale feature integration network (MFIN) to address the above issue. Specifically, to expand the receptive fields for global features, MFIN is constructed by cascading the multi-scale feature integration blocks (MFIBs) in a serial manner. Each MFIB contains a multi-scale feature extraction module (MFEM) and a feature integration unit (FIU). To enlarge the receptive fields at a granular level, the features in MFEM are cascaded in a parallel manner. To capture the full-image dependencies, FIU incorporates the dense and pixel-wise correlations from the outputs of MFEM efficiently. The conducted experiments demonstrate that our method outperforms state-of-the-art methods in quantitative and qualitative evaluation. Notably, the experimental results on running time state that our method can achieve real-time performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199 (2014)

  2. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654 (2016a)

  3. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645 (2016b)

  4. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144 (2017)

  5. Liu, F., Yu, Q., Chen, L., Jeon, G., Albertini, M.K., Yang, X.: Aerial image super-resolution based on deep recursive dense network for disaster area surveillance. Pers. Ubiquitous Comput. pp 1–10 (2021)

  6. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: IEEE conference on computer vision and pattern recognition (2017a)

  7. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European conference on computer vision (ECCV), pp 252–268 (2018)

  8. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 723–731 (2018)

  9. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM international conference on multimedia, pp 2024–2032 (2019)

  10. Gao, S., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.H.: Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence (2019)

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 (2016)

  12. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision, pp 4799–4807 (2017)

  13. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481 (2018b)

  14. Qiu, Y., Wang, R., Tao, D., Cheng, J.: Embedded block residual network: a recursive restoration model for single-image super-resolution. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4180–4189 (2019)

  15. Zhang, K., Gu, S., Timofte, R., Hui, Z., Wang, X., Gao, X., Xiong, D., Liu, S., Gang, R., Nan, N. et al.: Aim 2019 challenge on constrained super-resolution: Methods and results. In: 2019 IEEE/CVF international conference on computer vision workshop (ICCVW), IEEE, pp 3565–3574 (2019a)

  16. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution, pp 5835–5843 (2017)

  17. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: European conference on computer vision (2018)

  18. Hu, Y., Gao, X., Li, J., Huang, Y., Wang, H.: Single image super-resolution via cascaded multi-scale cross network. arXiv preprint arXiv:180208808 (2018b)

  19. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141 (2018a)

  20. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803 (2018)

  21. Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., Huang, T.S.: Ccnet: Criss-cross attention for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence (2020)

  22. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, Citeseer 30:3 (2013)

  23. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:13124400 (2013)

  24. Zhang, Y., Li, K., Li, K., Zhong, B., Fu, Y.: Residual non-local attention networks for image restoration. arXiv preprint arXiv:190310082 (2019b)

  25. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 126–135 (2017)

  26. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

  27. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International conference on curves and surfaces, Springer, pp 711–730 (2010)

  28. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE international conference on computer vision. ICCV 2001, IEEE, vol 2, pp 416–423 (2001)

  29. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5197–5206 (2015)

  30. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4), 600–612 (2004)

    Article  Google Scholar 

  31. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European conference on computer vision, Springer, pp 391–407 (2016)

  32. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision, pp 4539–4547 (2017b)

  33. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition (2018a)

  34. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  35. Chen, Y., Fan, H., Xu, B., Yan, Z., Kalantidis, Y., Rohrbach, M., Yan, S., Feng, J.: Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In: Proceedings of the IEEE international conference on computer vision, pp 3435–3444 (2019)

Download references

Acknowledgements

The research in our paper is sponsored by National Key R and D Program of China, under Grant No. 2020AAA0104500. The funding is from Sichuan University under grant 2020SCUNG205.

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, 610064, Sichuan, China

    Zheng He & Xiaomin Yang

  2. School of Aeronautics and Astronautics, Sichuan University, Chengdu, 610064, Sichuan, China

    Zheng He & Xiaomin Yang

  3. College of Electrical Engineering, Sichuan University, Chengdu, 610065, Sichuan, China

    Kai Liu

  4. TAL Education Group, Beijing, China

    Zitao Liu

  5. The Center of Gerontology and Geriatrics, West China Hospital, Sichuan University, Chengdu, China

    Qingyu Dou

Authors
  1. Zheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zitao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qingyu Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaomin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Kai Liu or Qingyu Dou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, Z., Liu, K., Liu, Z. et al. A lightweight multi-scale feature integration network for real-time single image super-resolution. J Real-Time Image Proc 18, 1221–1234 (2021). https://doi.org/10.1007/s11554-021-01142-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01142-7

Keywords

Search

Navigation