Skip to main content
SpringerLink
Log in

Real-time visible and near-infrared video fusion: architecture and implementation

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Near-infrared (NIR) band sensors capture achromatic images that contain complementary details of a scene which are diminished in visible (VS) band images when the scene is obscured by haze, mist, or fog. To exploit these complementary details, an integrated FPGA architecture and implementation of a video processing system are proposed in this paper. This system performs VS-NIR video fusion and produces an enhanced VS video in real-time. The proposed FPGA architecture and implementation effectively handle the challenges associated with the simultaneous processing of video signals obtained from different sources such as the inevitable delay among corresponding frames and time-varying deviation among frame rates. Moreover, the proposed implementation is efficiently designed and able to produce the fused video at the same frame rate as the input videos, i.e. in real-time, regardless of the resolution of the input videos while the consumed FPGA resources are kept small. This is achieved by data and calculations reuse, besides performing operations concurrently in parallel and pipelined fashions at both the data and task levels. The proposed implementation is synthesized, validated on a low-end FPGA device, and compared to three other implementations. The comparison shows the superiority of the proposed implementation in terms of the consumed resources which have a direct industrial impact in the case of integration in modern smart-phones and cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. The BT.656 format can also provide video data as a stream of words, where each word is 10 bits.

  2. We assume that the frame sizes of \({Y}^{\text {VS}}\) and \({Y}^{\text {NIR}}\) are equal.

  3. The video card presented in [7] is proposed by the same authors of this paper.

  4. To the best of our knowledge, the implemented fusion algorithm [1] does not have any hardware implementation available in the literature. So, the comparison might not be so conclusive because the compared implementations are for different algorithms.

References

  1. Awad, M., Elliethy, A., Aly, H.A.: Adaptive near-infrared and visible fusion for fast image enhancement. IEEE Trans. Comp. Imaging 6, 408–418 (2019)

    Article  Google Scholar 

  2. Colvero, C., Cordeiro, M., De Faria, G., Von Der Weid, J.: Experimental comparison between far-and near-infrared wavelengths in free-space optical systems. Microw. Opt. Technol. Lett. 46(4), 319–323 (2005)

    Article  Google Scholar 

  3. Bailey, D.G.: Design for Embedded Image Processing on FPGAs. Wiley, Hoboken (2011)

    Book  Google Scholar 

  4. Besiris, D., Tsagaris, V., Fragoulis, N., Theoharatos, C.: An FPGA-based hardware implementation of configurable pixel-level color image fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 362–373 (2012)

    Article  Google Scholar 

  5. Qu, F., Liu, B., Zhao, J., Sun, Q.: Image fusion real-time system based on FPGA and multi-DSP. Opt. Photonics J. 3(2), 76–78 (2013)

    Article  Google Scholar 

  6. Fredembach, C., Süsstrunk, S.: Colouring the near-infrared, in Proc. IS&T 16th Color Imag. Conf., Jan. 2008, pp. 176–182

  7. Awad, M., Abougindia, I.T., Elliethy, A., Aly, H.A.: Flexible architecture for real-time synchronized processing of multimedia signals. J. Multimedia Tools and Applications 80, 18531–18551 (2021)

    Article  Google Scholar 

  8. Jack, K.: Video demystified: a handbook for the digital engineer, 4th ed.   Elsevier, Sep. (2004)

  9. Radiocommunication Sector of ITU, BT.656: Interface for digital component video signals in 525-line and 625-line television systems operating at the 4:2:2 level of recommendation ITU-R BT.601, BT Series, Broadcasting service (television), Dec. 2007

  10. Tao, L., Ngo, H., Zhang, M., Livingston, A., Asari, V.: A multi-sensor image fusion and enhancement system for assisting drivers in poor lighting conditions, in 34th Applied Imagery and Pattern Recognition Workshop (AIPR’05)(AIPR), Oct. 2005, pp. 106–113

  11. Zhang, J., Han, Y., Chang, B., Yuan, Y., Qian, Y., Qiu, Y.: Real-time color image fusion for infrared and low-light-level cameras, in Proc. SPIE, vol. 7383, Aug. 2009, pp. 904 – 910

  12. Cai, L., Ning, Y., Diao, Y., Li, H., Li, G., Zhao, G.: A real time fusion system of infrared and low level light images base on FPGA, in 12th International Conference on Digital Image Processing (ICDIP), X. Jiang and H. Fujita, Eds., vol. 11519, June 2020, pp. 282–288

  13. Zhang, Z., Li, H., Zhao, G.: Bionic algorithm for color fusion of infrared and low light level image based on rattlesnake bimodal cells, IEEE Access, vol. 6, pp. 68 981–68 988, Nov (2018)

  14. Wielgus, M., Antoniewicz, A., Bartyś, M., Putz, B.: Fast and adaptive bidimensional empirical mode decomposition for the real-time video fusion, in 15th International Conference on Information Fusion (FUSION), July 2012, pp. 649–654

  15. Bhuiyan, S.M., Adhami, R.R., Khan, J.F.: A novel approach of fast and adaptive bidimensional empirical mode decomposition. In: IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp. 1313–1316 (2008)

  16. Bhuiyan, S.M., Adhami, R.R., Khan, J.F.: Fast and adaptive bidimensional empirical mode decomposition using order-statistics filter based envelope estimation, EURASIP J. Adv. Signal Process. 2008, no. 3, pp. 1–18, Jan. (2008)

  17. Putz, B., Bartyś, M., Antoniewicz, A., Klimaszewski, J., Kondej, M., Wielgus, M.: Real-time image fusion monitoring system: Problems and solutions, in Image Processing and Communications Challenges 4. Advances in Intelligent Systems and Computing, R. S. Choraś, Ed., vol. 184, 2013, pp. 143–152

  18. Awad, M., Elliethy, A., Aly, H. A.: A real-time FPGA implementation of visible/near infrared fusion based image enhancement, in Proc. IEEE Int. Conf. Image Process., Oct. 2018, pp. 3968–3972

  19. Jiang, J., Liu, C., Ling, S.: An FPGA implementation for real-time edge detection, J. Real-Time Image Process. July (2015)

  20. Choe, G., Kim, S., Im, S., Lee, J., Narasimhan, S.G., Kweon, I.S.: RANUS: RGB and NIR urban scene dataset for deep scene parsing. IEEE Trans. Robotics Automation 3(3), 1808–1815 (2018)

    Article  Google Scholar 

  21. DVD-V8000 Reference Level Professional DVD Player, Pioneer, 2008. [Online]. Available: https://www.pioneerelectronics.com/PUSA/Professional/Pro-Video/DVD-V8000

  22. Video capture, elgato, 2019. [Online]. Available: https://www.elgato.com/en/video-capture

  23. Elliethy, A., Aly, H.: Fast near infrared fusion-based adaptive enhancement of visible images, in Proc. IEEE Global Conf. Signal Inf. Process., Nov. 2017, pp. 156–160

  24. Radiocommunication Sector of ITU: Digital interfaces for studio signals with 1920\(\times\)1080 image formats. BT Series, Broadcasting service (television) (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Military Technical College, Cairo, Egypt

    Mohamed Awad, Ahmed Elliethy & Hussein A. Aly

Authors
  1. Mohamed Awad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed Elliethy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hussein A. Aly
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmed Elliethy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file3 (PDF 142 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Awad, M., Elliethy, A. & Aly, H.A. Real-time visible and near-infrared video fusion: architecture and implementation. J Real-Time Image Proc 18, 2479–2493 (2021). https://doi.org/10.1007/s11554-020-01068-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-01068-6

Keywords

Search

Navigation