Skip to main content

Advertisement

SpringerLink
Log in

Embedded real-time infrared and visible image fusion for UAV surveillance

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Infrared and visible image fusion is a beneficial processing task for Unmanned Aerial Vehicle (UAV) surveillance, which can improve visibility by combining the advantages of the infrared camera and the visible light camera. An embedded onboard solution is necessary for UAV-based surveillance missions because it reduces the amount of data that are transmitted to the ground. In this paper, we propose an infrared and visible light image fusion method and implement it on two platforms with commonly used HW accelerators for embedded vision applications: Zedboard (ARM + FPGA) and NVIDIA TX1 (ARM + GPU), and compare their performances. To verify the usefulness of image fusion, we carry out sufficient experiments to prove that image fusion can improve the target detection ability of a UAV in different scenes. The detection rate for target detection is up to 0.926 in our experiments. The execution times on the ZedBoard and the TX1 are, respectively, 205.3 FPS and 36.6 FPS (38 \(\times\) and 6.7 \(\times\) in comparison to an ARM Cortex-A9 processor). Our results also show that the ZedBoard achieves an energy/frame reduction ratio of 7.1 \(\times\) and 18.9 \(\times\) respectively compared to the TX1 and the ARM CPU. This work is based on a UAV platform designed by ourselves, and all image sets are real scenes that we have captured. This demonstrates that the proposed method is viable and reflects the actual needs of real UAV surveillance systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Dji matrice 600 pro - dji https://www.dji.com/ca/matrice600-pro

  2. Droneyee http://www.droneyee.com/

  3. Open source computer vision library, opencv. https://opencv.org/

  4. Sdsoc development environment - xilinx https://www.xilinx.com/products/design-tools/software-zone/sdsoc.html

  5. Using ip integrator - xilinx https://www.xilinx.com/products/design-tools/vivado/quicktake-videos/using-ip-integrator.html

  6. Vivado high-level synthesis https://www.xilinx.com/products/design-tools/vivado/integration/esl-design.html

  7. Xilinx opencv user guide https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_3/ug1233-xilinx-opencv-user-guide.pdf

  8. Xilinx revision webpage https://www.xilinx.com/products/design-tools/embedded-vision-zone.html

  9. Aydin, F., Ugurdag, H.F., Levent, V.E., Guzel, A.E., Annafianto, N.F.R., Ozkan, M.A., AkgUn, T., Erbas, C.: Rapid design of real-time image fusion on fpga using hls and other techniques. In: 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), pp. 1–6. IEEE (2018)

  10. Crockett, L.H., Elliot, R.A., Enderwitz, M.A., Stewart, R.W.: The Zynq Book: Embedded Processing with the Arm Cortex-A9 on the Xilinx Zynq-7000 All Programmable Soc. Strathclyde Academic Media (2014)

  11. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee (2009)

  12. Everingham, M., Winn, J.: The pascal visual object classes challenge 2012 (voc2012) development kit. Tech. Rep, Pattern Analysis, Statistical Modelling and Computational Learning (2011)

  13. Fendri, E., Boukhriss, R.R., Hammami, M.: Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal. Appl. 20(4), 907–926 (2017)

    Article  MathSciNet  Google Scholar 

  14. Georgis, G., Lentaris, G., Reisis, D.: Acceleration techniques and evaluation on multi-core cpu, gpu and fpga for image processing and super-resolution. J. Real Time Image Process. 16(4), 1207–1234 (2019)

    Article  Google Scholar 

  15. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp. 1440–1448 (2015)

  16. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2012)

    Article  Google Scholar 

  17. Hutchinson, S., Hager, G.D., Corke, P.I.: A tutorial on visual servo control. IEEE Trans. Robot. Autom. 12(5), 651–670 (1996)

    Article  Google Scholar 

  18. Jabbari, H., Oriolo, G., Bolandi, H.: An adaptive scheme for image-based visual servoing of an underactuated uav. Int. J. Robot. Autom. 29(1), 92–104 (2014)

    Google Scholar 

  19. Jin, X., Jiang, Q., Yao, S., Zhou, D., Nie, R., Hai, J., He, K.: A survey of infrared and visual image fusion methods. Infrar. Phys. Technol. 85, 478–501 (2017)

    Article  Google Scholar 

  20. Kanellakis, C., Nikolakopoulos, G.: Survey on computer vision for uavs: current developments and trends. J. Intell. Robot. Syst. 87(1), 141–168 (2017)

    Article  Google Scholar 

  21. Li, S., Kang, X., Fang, L., Hu, J., Yin, H.: Pixel-level image fusion: a survey of the state of the art. Inform. Fus. 33, 100–112 (2017)

    Article  Google Scholar 

  22. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European conference on computer vision, pp. 21–37. Springer (2016)

  23. Qasaimeh, M., Denolf, K., Lo, J., Vissers, K., Zambreno, J., Jones, P.H.: Comparing energy efficiency of cpu, gpu and fpga implementations for vision kernels. In: 2019 IEEE International Conference on Embedded Software and Systems (ICESS), pp. 1–8. IEEE (2019)

  24. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788 (2016)

  25. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp. 91–99 (2015)

  26. Shah, P., Merchant, S., Desai, U.B.: Fusion of surveillance images in infrared and visible band using curvelet, wavelet and wavelet packet transform. Int. J. Wavel. Multiresolut. Inform. Process. 8(02), 271–292 (2010)

    Article  MathSciNet  Google Scholar 

  27. Sun, P., Achim, A., Hasler, I., Hill, P., Nunez-Yanez, J.: Energy efficient video fusion with heterogeneous cpu-fpga devices. In: 2016 Design, Automation and Test in Europe Conference and Exhibition (DATE), pp. 1399–1404. IEEE (2016)

  28. Zhao, B., Li, Z., Liu, M., Cao, W., Liu, H.: Infrared and visible imagery fusion based on region saliency detection for 24-h-surveillance systems. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1083–1088. IEEE (2013)

Download references

Acknowledgements

The authors acknowledge the State Key Laboratory of High Performance Computing, National University of Defense Technology, P.R. China. The support provided by the China Scholarship Council (CSC) and Dusan and Anne Miklas Chair for Engineering Design of the University of Toronto is gratefully acknowledged.

Author information

Authors and Affiliations

  1. College of Computer, National University of Defense Technology, Changsha, China

    Jun Li & Yuanxi Peng

  2. College of Advanced Interdisciplinary Studies, National University of Defense Technology, Changsha, China

    Tian Jiang

  3. Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada

    Jun Li

Authors
  1. Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanxi Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tian Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuanxi Peng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Peng, Y. & Jiang, T. Embedded real-time infrared and visible image fusion for UAV surveillance. J Real-Time Image Proc 18, 2331–2345 (2021). https://doi.org/10.1007/s11554-021-01111-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01111-0

Keywords

Search

Navigation