Skip to main content
SpringerLink
Log in

An FPGA-based design for real-time super-resolution reconstruction

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

For several decades, the camera spatial resolution is gradually increasing with the CMOS technology evolution. The image sensors provide more and more pixels, generating new constraints for suitable optics. As an alternative, promising solutions propose super-resolution (SR) techniques to reconstruct high-resolution images or video without modifying the sensor architecture. However, most of the SR implementations are far from reaching real-time performance on a low-budget hardware platform. Moreover, convincing state-of-the-art studies reveal that artifacts can be observed in highly textured areas of the image. In this paper, we propose a local adaptive spatial super resolution (LASSR) method to fix this limitation. LASSR is a two-step SR method including a machine learning-based texture analysis and a fast interpolation method that performs a pixel-by-pixel SR. Multiple evaluations of our method are also provided using standard image metrics for quantitative evaluation and also a psycho-visual assessment for a perceptual evaluation. A first FPGA-based implementation of the proposed method is then presented. It enables high-quality 2–4 k super-resolution videos to be performed at 16 fps, using only 13% of the FPGA capacity, opening the way to reach more than 60 fps by executing several parallel instances of the LASSR code on the FPGA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Arivazhagan, S., Ganesan, L.: Texture classification using wavelet transform. Pattern Recogn. Lett. 24(9–10), 1513–1521 (2003)

    Article  Google Scholar 

  2. Baker, S., Kanade, T.: Hallucinating faces. In: Proceedings 4th IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580), pp. 83–88 (2000)

  3. Borman, S., Stevenson, R.L.: Super-resolution from image sequences-a review. In: 1998 Midwest symposium on circuits and systems (Cat. No. 98CB36268), pp. 374–378 (1998)

  4. Bowen, Oliver, Bouganis, Christos-Savvas: real-time image super resolution using an FPGA. In: 2008 International conference on field programmable logic and applications, pp. 89–94 (2008)

  5. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  6. Chan, R.H., Chan, T.F., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24, 1408–1432 (2003)

    Article  MathSciNet  Google Scholar 

  7. Chang, C.-C., Lin, C.-J.: Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011)

    Article  Google Scholar 

  8. Choi, J.-S., Kim, M.: Super-interpolation with edge-orientation-based mapping Kernels for low complex 2 x upscaling. IEEE Trans. Image Process. 25, 469–483 (2016)

    Article  MathSciNet  Google Scholar 

  9. Coomans, D., Massart, D.L.: Alternative k-nearest neighbour rules in supervised pattern recognition: part 1. K-nearest neighbour classification by using alternative voting rules. Anal. Chim. Acta 136, 15–27 (1982)

    Article  Google Scholar 

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer vision—ECCV 2014, 13th European conference, Zurich, pp. 184–199 (2014)

  11. Dubois, J., Mattavelli, M.: Embedded co-processor architecture for CMOS based image acquisition. In: Image processing, 2003. Proceedings 2003 international conference on, volume 2, IEEE, pp. II–591 (2003)

  12. Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14, 47–57 (2004)

    Article  Google Scholar 

  13. Feichtenhofer, C., Fassold, H., Schallauer, P.: A perceptual image sharpness metric based on local edge gradient analysis. IEEE Signal Process. Lett. 20(4), 379–382 (2013)

    Article  Google Scholar 

  14. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 56–65 (2002)

    Article  Google Scholar 

  15. Freund, Y., Schapire, R., Abe, N.: A short introduction to boosting. J. Jpn. Soc. Artif. Intell. 14(771–780), 1612 (1999)

    Google Scholar 

  16. Georgis, G., Lentaris, G., Reisis, D.: Acceleration techniques and evaluation on multi-core CPU, GPU and FPGA for image processing and super-resolution. J. Real-Time Image Process. pp. 1–28 (2016)

  17. Gohshi, S.: A new signal processing method for video image-reproduce the frequency spectrum exceeding the Nyquist frequency using a single frame of the video image. Springer, New York (2013)

  18. Guan, J., Yang, J., Huang, Y., Li, W.: Maximum a posteriori-based angular superresolution for scanning radar imaging. IEEE Trans. Aerosp. Electron. Syst. 50, 2389–2398 (2014)

    Article  Google Scholar 

  19. Haralick, R.M.: Statistical and structural approaches to texture. Proc. IEEE 67(5), 786–804 (1979)

    Article  Google Scholar 

  20. He, Z., Huang, H., Jiang, M., Bai, Y., Luo, G.: FPGA-based real-time super-resolution system for ultra high definition videos. In: 2018 IEEE 26th annual international symposium on field-programmable custom computing machines (FCCM), Boulder, CO, pp. 181–188 (2018)

  21. Jung, C., Ke, P., Sun, Z., Aiguo, G.: A fast deconvolution-based approach for single-image super-resolution with GPU acceleration. J. Real-Time Image Proc. 14, 501–512 (2018)

    Article  Google Scholar 

  22. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1646–1654 (2016)

  23. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1637–1645 (2016)

  24. Kim, Y., Choi, J.-S., Kim, M.: 2X super-resolution hardware using edge-orientation-based linear mapping for real-time 4K UHD 60 fps video applications. IEEE Trans. Circ. Syst. II Express Briefs 65, 1274–1278 (2018)

    Google Scholar 

  25. Kim, Y., Choi, J.-S., Kim, M.: A real-time convolutional neural network for super-resolution on FPGA with applications to 4K UHD 60 fps video services. IEEE Trans. Circ. Syst. Video Technol. pp. 1–1 (2018)

  26. Kronsteiin, D.: Video stream scaler. www.opencores.org (2011)

  27. Li, H., Huang, Y., Kuang, C., Liu, X.: Method of super-resolution based on array detection and maximum-likelihood estimation. Appl. Opt. 55, 9925 (2016)

    Article  Google Scholar 

  28. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp. 1132–1140 (2017)

  29. Madhukar, B.N., Narendra, R.: Lanczos resampling for the digital processing of remotely sensed images. In: V.S. Chakravarthi, Yasha Jyothi M.S., Rekha P. (eds), Proceedings of international conference on VLSI, communication, advanced devices, signals and systems and networking (VCASAN-2013), Springer, India, pp. 403–411 (2013)

  30. Manabe, T., Shibata, Y., Oguri, K.: FPGA implementation of a real-time super-resolution system using a convolutional neural network. In: 2016 International conference on field-programmable technology (FPT), Xi’an, pp. 249–252 (2016)

  31. Martin, Č: Perceptually based image quality assessment and image transformations. Ph.d. thesis, Department of Computer Science and Engineering, Faculty of Electrical Engineering, Czech Technical University in Prague (2008)

  32. Matsuyama, T., Miura, S.-I., Nagao, M.: Structural analysis of natural textures by Fourier transformation. Comput. Vis. Graph Image Process. 24(3), 347–362 (1983)

    Article  Google Scholar 

  33. Mitéran, J., Jiri Matas, E., Bourennane, M.P., Dubois, J.: Automatic hardware implementation tool for a discrete adaboost-based decision algorithm. EURASIP J. Appl. Sig. Process. 2005, 1035–1046 (2005)

    MATH  Google Scholar 

  34. Mosqueron, R., Dubois, J., Mattavelli, M., Mauvilet, D.: Smart camera based on embedded hw/sw coprocessor. EURASIP J. Embedded Syst. 2008(1), 597872 (2008)

    Article  Google Scholar 

  35. Namboodiri, V.P., De Smet, V., Van Gool, L.: Systematic evaluation of super-resolution using classification. In: 2011 Visual communications and image processing (VCIP), IEEE, pp. 1–4 (2011)

  36. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25, 1423–1468 (2014)

    Article  Google Scholar 

  37. Nguyen, K, Fookes, C., Sridharan, S., Tistarelli, M., Nixon, M.: Super-resolution for biometrics: a comprehensive survey. Pattern Recognit. 78, 23–42 (2018)

  38. van Ouwerkerk, J.D.: Image super-resolution survey. Image Vis. Comput. 24, 1039–1052 (2006)

    Article  Google Scholar 

  39. Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Magn. 20, 21–36 (2003)

    Article  Google Scholar 

  40. Pérez, J., Magdaleno, E., Pérez, F., Rodríguez, M., Hernández, D., Corrales, J.: Super-resolution in plenoptic cameras using FPGAs. Sensors 14, 8669–8685 (2014)

    Article  Google Scholar 

  41. Qin, F., He, X., Chen, W., Yang, X., Wei, W.: Video super resolution reconstruction based on subpixel registration and iterative back projection. J. Electron. Imaging 18, 013007 (2009)

  42. Quevedo, E., Sánchez, L., Callicó, G.M., Tobajas, F., de la Cruz, J., de Armas, V., Sarmiento, R.: Super-resolution with selective filter based on adaptive window and variable macro-block size. J. Real Time Image Proc. 15, 389–406 (2018)

    Article  Google Scholar 

  43. Redlich, R., Araneda, L., Saavedra, A., Figueroa, M.: An embedded hardware architecture for real-time super-resolution in infrared cameras. In: 2016 Euromicro conference on digital system design (DSD), pp. 184–191 (2016)

  44. Rhee, S., Kang, M.G.: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Opt. Eng. 38, 1348–1356 (1999)

    Article  Google Scholar 

  45. Robinson, M.D., Chiu, S.J., Toth, C.A., Izatt, J.A., Lo, J.Y., Farsiu, S.: New applications of super-resolution in medical imaging. In: Milanfar, P. (ed.) Super-resolution imaging, pp. 383–412. CRC Press, Boca Raton (2010)

    Google Scholar 

  46. Sajjadi, M.S.M., Scholkopf, B., Hirsch, M.: Enhancement: single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE international conference on computer vision, pp. 4491–4500 (2017)

  47. Seyid, K., Blanc, S., Leblebici, Y.: Hardware implementation of real-time multiple frame super-resolution. In: 2015 IFIP/IEEE international conference on very large scale integration (VLSI-SoC), Daejeon, pp. 219–224 (2015)

  48. Shen, W., Fang, L., Chen, X., Xu, H.: Projection onto convex sets method in space-frequency domain for super resolution. J. Comput. 9(8), 1959–1966 (2014)

  49. Shocher, A., Cohen, N., Irani, M.: Zero-shot super-resolution using deep internal learning. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp. 3118–3126 (2018)

  50. Tsai, R.Y., Huang, T.S.: Multiframe image restoration and registration. In: Tsai, R.Y., Huang, T.S. (eds.) Advances in Computer Vision and Image Processing, pp. 317–339. JAI Press Inc., Stamford (1984)

    Google Scholar 

  51. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Berlin (2013)

    MATH  Google Scholar 

  52. Wang, Z., Bovik, H.R., Sheikh, A.C., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  53. Wang, Z., Chen, J., Steven, C.H.H.: Deep learning for image super-resolution: a survey (2019)

  54. Yang, C.-Y., Ma, C., Yang, M.-H. (2014) Single-image super-resolution: a benchmark. In: David, F., Tomas, P., Bernt, S., Tinne, T. (eds), Computer Vision—ECCV 2014, Springer, Cham, pp. 372–386 (2014)

  55. Yang, D., Li, Z., Xia, Y., Chen, Z.: Remote sensing image super-resolution: challenges and approaches. In: 2015 IEEE international conference on digital signal processing (DSP), pp. 196–200 (2015)

  56. Yang, J., Huang, T.: Image super-resolution: historical overview and future challenges. In: Milanfar, P. (ed.) Super-Resolution Imaging, pp. 1–33. CRC Press, Boca Raton (2010)

    Google Scholar 

  57. Yuan, Y., Yang, X., Wei, W., Li, H., Liu, Y., Liu, K.: A fast single-image super-resolution method implemented with CUDA. J. Real-Time Image Proc. 16, 81–97 (2019)

    Article  Google Scholar 

  58. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Sig. Process. 128, 389–408 (2016)

    Article  Google Scholar 

  59. Zhang, G.P.: Neural networks for classification: a survey. IEEE Trans. Syst. Man. Cybern. Part C (Appl. Rev.) 30(4), 451–462 (2000)

    Article  Google Scholar 

  60. Zhang, L., Zhang, H., Shen, H., Li, P.: A super-resolution reconstruction algorithm for surveillance images. Sig. Process. 90, 848–859 (2010)

    Article  Google Scholar 

  61. Zhou, F., Yao, R., Liu, B., Qiu, G.: Visual quality assessment for super-resolved images: database and method. IEEE Trans. Image Process. pp. 1–1 (2019)

Download references

Acknowledgements

This work was funded by EU Horizon 2020 ECSEL JU research and innovation program under Grant agreement 662222 (EXIST).

Author information

Authors and Affiliations

  1. Université Bourgogne Franche-Comté (UBFC), 9 Avenue Alain Savary, 21000, Dijon, France

    Yoan Marin, Johel Miteran, Julien Dubois, Barthélémy Heyrman & Dominique Ginhac

Authors
  1. Yoan Marin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johel Miteran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Julien Dubois
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Barthélémy Heyrman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dominique Ginhac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dominique Ginhac.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marin, Y., Miteran, J., Dubois, J. et al. An FPGA-based design for real-time super-resolution reconstruction. J Real-Time Image Proc 17, 1769–1785 (2020). https://doi.org/10.1007/s11554-020-00944-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00944-5

Keywords

Search

Navigation