Skip to main content
SpringerLink
Log in

Single-image super-resolution based on multi-branch residual pyramid network

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

The convolutional neural network plays an important role in single-image super-resolution reconstruction. In this study, we presented a branch residual pyramid channel network to reconstruct high-resolution image from single low-resolution image. Specifically, the low-level information was reconstructed based on the low-resolution image features. Then, we completed the characteristics of the advanced information mapping of the residual neural network. A different number of serial sequence and cross-combined convolution repeatedly were applied to strengthen the key information of mining capacity which increased the multi-branch structure while kept the direct path, we used. In addition, a feature pyramid channel attention module (feature maps aggregated through a top–down path) in each horizontal connection was utilized which generate more targeted feature maps. Experimental results with extensive quantitative and qualitative evaluation on benchmark data sets demonstrated that the proposed algorithm is not only superior to the existing advanced algorithms in speed and accuracy, but also shows good effect in super-resolution reconstruction of single image. Specifically, performance of the proposed method is particularly outstanding on SET5 data set (PSNR value is 0.08 higher than that of the Subprime network), SET14 data set (PSNR value is 0.03 higher than that of the Subprime network) and Urban100 data set (PSNR value is 0.15 higher than that of the Subprime network) when the scaling factor is 2, 3 and 4, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Zhang, Y., Fan, Q., Bao, F., Liu, Y., Zhang, C.: Single-image super-resolution based on rational fractal interpolation. IEEE Trans. Image Process. 27(8), 3782–3797 (2018)

    Article  MathSciNet  Google Scholar 

  2. Shivagunde, S., and Biswas, M.: Single image super-resolution based on modified interpolation method using MLP and DWT. In 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), pp. 212–219 (2019)

  3. Dong, C., Loy, C. C., He, K., and Tang, X.: Learning a deep convolutional network for image super-resolution. In European conference on computer vision, pp. 184–199 (2014)

  4. Kim, J., Kwon Lee, J., and Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.1646–1654 (2016)

  5. Chang, H., Yeung, D. Y., Xiong, Y.:Super-resolution through neighbor embedding. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. I–I (2004)

  6. Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM Trans. Graph. (TOG) 30(2), 1–11 (2011)

    Article  Google Scholar 

  7. Xiong, Z., Sun, X., Wu, F.: Robust web image/video super-resolution. IEEE Trans. Image Process. 19(8), 2017–2028 (2010)

    Article  MathSciNet  Google Scholar 

  8. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  9. Dong, C., Loy, C. C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European conference on computer vision (ECCV), pp. 391–407 (2016)

  10. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., and Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874–1883 (2016)

  11. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: European Conference on Computer Vision (ECCV), pp. 517–532 (2018)

  12. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision (ECCV), pp. 286–301 (2018)

  13. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3147–3155 (2017)

  14. Lai, W. S., Huang, J. B., Ahuja, N., Yang, M. H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 624–632(2017)

  15. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690 (2017)

  16. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 136–144(2017)

  17. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141 (2018)

  18. Roy, A.G., Navab, N., Wachinger, C.: Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks. IEEE Trans. Med. Imag. 38(2), 540–549 (2018)

    Article  Google Scholar 

  19. Woo, S., Park, J., Lee, J. Y., Kweon, I. S.: Cbam: Convolutional block attention module. In: European conference on computer vision (ECCV), pp. 3–19 (2018)

  20. Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: A2-Nets: double attention networks. arXiv preprint arXiv:1810.11579 (2018)

  21. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146–3154 (2019)

  22. He, H., Li, Z., Li, J., Peng, X.: Image super-resolution through pyramid learning. In 2012 Fourth International Conference on Digital Home, pp. 241–245 (2012)

  23. Zhang, S., Zhang, H., Zhang, L., Xue, Y.: A multi-scale-based super-resolution method for face image. In 2010 3rd International Congress on Image and Signal Processing, pp. 587–590 (2010)

  24. Mo, Z., He, X., Li, G., Chen, J.: Ladder pyramid networks for single image super-resolution. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 578–582 (2020)

  25. Zhao, T., Wu, X.: Pyramid feature attention network for saliency detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3085–3094 (2019)

  26. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T. S.: Sca-cnn: spatial and channel-wise attention in convolutional networks for image captioning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5659–5667 (2017)

  27. Gao, X., Sun, X., Yan, M., Sun, H., Fu, K., Zhang, Y., Ge, Z.: Road extraction from remote sensing images by multiple feature pyramid network. In IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 6907–6910 (2018)

  28. Zhao, Y., Han, R., Rao, Y.: A new feature pyramid network for object detection. In 2019 International Conference on Virtual Reality and Intelligent Systems (ICVRIS), pp. 428–431 (2019)

  29. Yin, S., Xin, J., Wang, Y., Basu, A.: Image dehazing with uneven illumination prior by dense residual channel attention network. IET Image Proc. 14(13), 3260–3272 (2020)

    Article  Google Scholar 

  30. Zheng, K., Gao, L., Zhang, B., Cui, X.: Multi-losses function based convolution neural network for single hyperspectral image super-resolution. In 2018 Fifth International Workshop on Earth Observation and Remote Sensing Applications (EORSA), pp. 1–4 (2018)

  31. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.126–135 (2017)

  32. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M. L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012) https://doi.org/10.5244/C.26.135

  33. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In International conference on curves and surfaces, pp. 711–730 (2010)

  34. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (ICCV), pp. 416–423 (2001)

  35. Huang, J. B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015)

  36. Baig, M. A., Moinuddin, A. A., Khan, E.: PSNR of highest distortion region: an effective image quality assessment method. In 2019 International Conference on Electrical, Electronics and Computer Engineering (UPCON), pp. 1–4 (2019)

  37. Xia, B., Cao, J., Wang, C.: SSIM-NET: Real-time PCB defect detection based on SSIM and MobileNet-V3. In 2019 2nd World Conference on Mechanical Engineering and Intelligent Manufacturing (WCMEIM), pp. 756–759 (2019)

  38. Kim, H., Kim, G.: Single image super-resolution using fire modules with asymmetric configuration. IEEE Signal Process. Lett. 27, 516–519 (2020)

    Article  Google Scholar 

  39. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019)

  40. Shi, Y., Li, S., Li, W., Liu, A.: Fast and lightweight image super-resolution based on dense residuals two-channel network. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2826–2830 (2019)

  41. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3271 (2018)

  42. Li, J., Fang, F., Li, J., Mei, K., Zhang, G.: MDCN: Multi-scale Dense Cross Network for Image Super-Resolution. IEEE Trans. Circuits Syst. Video Technol. 31(7), 2547–2561 (2020)

    Article  Google Scholar 

  43. He, Z., Cao, Y., Du, L., Xu, B., Yang, J., Cao, Y., Tang, S., Zhuang, Y.: Mrfn: multi-receptive-field network for fast and accurate single image super-resolution. IEEE Trans. Multimed 22(4), 1042–1054 (2019)

    Article  Google Scholar 

  44. Fang, F., Li, J., Zeng, T.: Soft-edge assisted network for single image super-resolution. IEEE Trans. Image Process. 29, 4656–4668 (2020)

    Article  Google Scholar 

  45. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  46. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3791–3799 (2015)

  47. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE international conference on computer vision, pp. 370–378 (2015)

  48. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In Asian conference on computer vision, pp. 111–126(2014)

  49. Baek, S., Lee, C.: Single image super-resolution using frequency-dependent convolutional neural networks. In 2020 IEEE International Conference on Industrial Technology (ICIT), pp. 692–695 (2020)

  50. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144(2017)

  51. He, X., Mo, Z., Wang, P., Liu, Y., Yang, M., Cheng, J.: Ode-inspired network design for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1732–1741 (2019)

  52. Esmaeilzehi, A., Ahmad, M.O., Swamy, M.N.S.: SRNSSI: A Deep Light-Weight Network for Single Image Super Resolution Using Spatial and Spectral Information. IEEE Trans. Comput. Imaging 7, 409–421 (2021)

    Article  Google Scholar 

  53. Yang, X., Xie, T., Liu, L., Zhou, D.: Image super-resolution reconstruction based on improved Dirac residual network. Multidim. Syst. Signal Process., pp. 1–18 (2021)

Download references

Author information

Authors and Affiliations

  1. Shandong Normal University, Jinan, China

    Jiayu Ou, Hao Xia, Wenxiao Huo, Yejin Yan & Tianping Li

Authors
  1. Jiayu Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenxiao Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yejin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tianping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tianping Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ou, J., Xia, H., Huo, W. et al. Single-image super-resolution based on multi-branch residual pyramid network. J Real-Time Image Proc 18, 2569–2581 (2021). https://doi.org/10.1007/s11554-021-01150-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01150-7

Keywords

Search

Navigation