Skip to main content
SpringerLink
Log in

Lightweight image super-resolution network using involution

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Recently, the single image super-resolution methods with deep and complex convolutional neural network structures have achieved remarkable performance. However, those approaches improve the performance at the cost of higher memory occupation, which are difficult to be applied for some resource-constrained devices. With the goal of minimizing parameters, an effective and efficient operator named involution is introduced in our proposed model, delivering enhanced performance at reduced cost compared to convolution-based counterparts. On the basis of involution, we propose two building blocks named RMFDB(Residual Mixed Feature Distillation Block) and CICB(Conv-Invo-Conv Block) for the main module and the reconstruction module respectively. RMFDB has the similar structure as the RFDB but with our involution layers. This block is much more lightweight and efficient than conventional convolution-based blocks. CICB combines the nearest-neighbor upsampling, convolution and involution layers. The final reconstruction quality is improved with little parameter cost. Experimental results demonstrate the effectiveness of the proposed model against the state-of-the-art (SOTA) SR methods. Our final model could achieve similar performance as the lightweight networks RFDN and PAN, but with only 224K parameters and 64.2G Multi-Adds with the scale factor of 2. The effectiveness of each proposed components is also validated by ablation study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bates, M., Huang, B., Dempsey, G.T., Zhuang, X.: Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science 317(5845), 1749–1753 (2007)

    Article  Google Scholar 

  2. Huang, B., Wang, W., Bates, M., Zhuang, X.: Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319(5864), 810–813 (2008)

    Article  Google Scholar 

  3. Hamaide, J., De Groof, G., Van Steenkiste, G., Jeurissen, B., Van Audekerke, J., Naeyaert, M., Van Ruijssevelt, L., Cornil, C., Sijbers, J., Verhoye, M., Van der Linden, A.: Exploring sex differences in the adult zebra finch brain: In vivo diffusion tensor imaging and ex vivo super-resolution track density imaging. NeuroImage 146, 789–803 (2017)

    Article  Google Scholar 

  4. Jurek, J., Kociński, M., Materka, A., Elgalal, M., Majos, A.: Cnn-based superresolution reconstruction of 3d mr images using thick-slice scans. Biocybern. Biomed. Eng. 40(1), 111–125 (2020)

    Article  Google Scholar 

  5. Thornton, M.W., Atkinson, P.M., Holland, D.A.: Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int. J. Remote Sens. 27(3), 473–491 (2006)

    Article  Google Scholar 

  6. Gunturk, B.K., Batur, A.U., Altunbasak, Y., Hayes, M.H., Mersereau, R.M.: Eigenface-domain super-resolution for face recognition. IEEE Trans. Image Process. 12(5), 597–606 (2003)

    Article  Google Scholar 

  7. Zhang, L., Zhang, H., Shen, H., Li, P.: A super-resolution reconstruction algorithm for surveillance images. Signal Process. 90(3), 848–859 (2010)

    Article  Google Scholar 

  8. Zhang, L., Xiaolin, W.: An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15(8), 2226–2238 (2006)

    Article  Google Scholar 

  9. Zhang, K., Gao, X., Tao, D., Li, X.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21(11), 4544–4556 (2012)

    Article  MathSciNet  Google Scholar 

  10. Lan, R., Zhou, Y., Liu, Z., Luo, X.: Prior knowledge-based probabilistic collaborative representation for visual recognition. IEEE Trans. Cybern. 50(4), 1498–1508 (2020)

    Article  Google Scholar 

  11. Li, B., Liu, R., Cao, J., Zhang, J., Lai, Y.-K., Liu, Xiuping: Online low-rank representation learning for joint multi-subspace recovery and clustering. IEEE Trans. Image Process. 27(1), 335–348 (2018)

    Article  MathSciNet  Google Scholar 

  12. Timofte, R., Rothe, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1865–1873, (2016)

  13. Yang, J., Wright, J., Huang, T.S., Ma, Yi.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  14. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144, (2017)

  15. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301, (2018)

  16. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11057–11066, (2019)

  17. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June (2020)

  18. Chao, D., Chen, C.L., Kaiming, H., Xiaoou, T.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Google Scholar 

  19. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654, (2016)

  20. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481, (2018)

  21. Li, D., Hu, J., Wang, C., Li, X., She, Q., Zhu, L., Zhang, T., Chen, Q.: Involution: Inverting the inherence of convolution for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12321–12330, (2021)

  22. Alex, K., Ilya, S., Geoffrey, E.H.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  23. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprintarXiv:1704.04861, (2017)

  24. Liu, J.-J., Hou, Q., Cheng, M.-M., Wang, C., Feng, J.: Improving convolutional networks with self-calibrated convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10096–10105, (2020)

  25. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848–6856, (2018)

  26. Ahn, N., Kang, B., Sohn, K.-A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 252–268, (2018)

  27. Hui, Z., Gao, X., Wang, X.: Lightweight image super-resolution with feature enhancement residual network. Neurocomputing 404, 50–60 (2020)

    Article  Google Scholar 

  28. Fan, Y., Yu, J., Huang, T.S.: Wide-activated deep residual networks based restoration for bpg-compressed images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2621–2624, (2018)

  29. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520, (2018)

  30. Huang, H., Shen, L., He, C., Dong, W., Huang, H., Shi, G.: Lightweight image super-resolution with hierarchical and differentiable neural architecture search. arXiv preprintarXiv:2105.03939, (2021)

  31. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprintarXiv:1511.07122, (2015)

  32. Zhao, H., Kong, X., He, J., Qiao, Y., Dong, C.: Efficient image super-resolution using pixel attention. In European Conference on Computer Vision, pp. 56–72. Springer, (2020)

  33. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 723–731, (2018)

  34. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multidistillation network. In ACM Multimedia, Xiumei (2019)

  35. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In European Conference on Computer Vision, pp 41–55. Springer, (2020)

  36. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y. Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0–0, (2018)

  37. Zhou W., Alan C B., Hamid R S., and Eero P S.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, (2004)

  38. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel: Low-complexity single-image super-resolution based on nonnegative neighbor embedding, Marie Line (2012)

  39. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  40. Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Iccv Vancouver: (2001)

  41. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5197–5206, (2015)

  42. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl 76(20), 21811–21838 (2017)

    Article  Google Scholar 

  43. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3147–3155, (2017)

  44. Diederik P K.and Jimmy B. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, (2014)

  45. Paszke, A., Gross, S., Chintala, S.,, Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer. Automatic differentiation in pytorch, Adam (2017)

  46. Dong, C., Loy, C. C., Tang, X.: Accelerating the super-resolution convolutional neural network. In European conference on computer vision, pp. 391–407. Springer, (2016)

  47. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 624–632, (2017)

  48. Wang, X., Wang, Q., Zhao, Y., Yan, J., Fan, L., Chen, L.: Lightweight single-image super-resolution network with attentive auxiliary feature learning. In Proceedings of the Asian Conference on Computer Vision, (2020)

  49. Chu, X., Zhang, B. Ma, H., Xu, R., Li, Q.: Fast, accurate and lightweight super-resolution with neural architecture search. In 2020 25th International Conference on Pattern Recognition (ICPR), pp 59–64. IEEE, (2021)

  50. Wang, C., Li, Z., Shi, J.: Lightweight image super-resolution with adaptive weighted learning network. arXiv preprintarXiv:1904.02358, (2019)

  51. Yan, Y., Xue, X., Chen, W., Peng, X.: Lightweight attended multi-scale residual network for single image super-resolution. IEEE Access 9, 52202–52212 (2021)

    Article  Google Scholar 

  52. Sun, L., Liu, Z., Sun, X., Liu, L., Lan, R., Luo, X.: Lightweight image super-resolution via weighted multi-scale residual network. IEEE/CAA J. Automatica Sinica 8(7), 1271–1280 (2021)

    Article  Google Scholar 

  53. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3867–3876, (2019)

  54. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision, pp. 4539–4547, (2017)

  55. Lan, R., Sun, L., Liu, Z., Huimin, L., Pang, C., Luo, X.: Madnet: A fast and lightweight network for single-image super resolution. IEEE Trans. Cybern. 51(3), 1443–1453 (2021)

    Article  Google Scholar 

  56. Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., Shah, M.: Transformers in vision: A survey. ACM Computing Surveys (CSUR), (2021)

  57. Wang, Z., Gao, G., Li, J., Yu, Y., Lu, H.: Lightweight image super-resolution with multi-scale feature interaction network. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, (2021)

Download references

Acknowledgements

This work is supported by National key research and development program (2019YFC1521105), Key R & D and transformation plan Qinghai Province (2020-GX-110, 2022-SF-140)

Author information

Authors and Affiliations

  1. School of Computer Science, Shaanxi Normal University, Xi’an, 710119, China

    Jiu Liang, Yu Zhang, Jiangbo Xue, Yu Zhang & Yanda Hu

Authors
  1. Jiu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiangbo Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanda Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, J., Zhang, Y., Xue, J. et al. Lightweight image super-resolution network using involution. Machine Vision and Applications 33, 68 (2022). https://doi.org/10.1007/s00138-022-01307-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-022-01307-9

Keywords

Search

Navigation