Review
Infrared machine vision and infrared thermography with deep learning: A review

https://doi.org/10.1016/j.infrared.2021.103754Get rights and content

Abstract

Infrared imaging-based machine vision (IRMV) is the technology used to automatically inspect, detect, and analyse infrared images (or videos) obtained by recording the intensity of infrared light emitted or reflected by observed objects. Depending on whether controllable excitation is used during the imaging of infrared rays, thermal IRMV can be categorised into passive thermography and active thermography. Passive thermography is an important supplement to conventional machine vision based on visible light and is a valid imaging tool for self-heating objects such as the human body and electrical power devices. Active thermography is a non-destructive testing method for the quality evaluation and safety assurance of non-self-heating objects. In active thermography, the trend is to inspect rapidly, reliably, and intelligently by introducing multiple-mode excitation sources and artificial intelligence. The rapid development of deep learning makes IRMV more and more intelligent and highly automated, thus considerably increasing its range of applications. This paper reviews the principle, cameras, and thermal data of IRMV and discusses the applications of deep learning applied to IRMV. Case studies of IRMV and deep learning on various platforms such as unmanned vehicles, mobile phones and embedded systems are also reported.

Introduction

Human eyes are sensitive to electromagnetic rays with wavelengths ranging from 400 to 760 nm (i.e. the visible light, VL). Machine vision (MV) is the technology that enables imaging machines to inspect, recognize and analyse images as human beings. A typical MV-based system is shown in Fig. 1(a), which includes VL cameras with lenses, light sources, personal computers (PCs, or embedded system) for image processing and actuators. Couple charged devices (CCDs) and complementary metal oxide semiconductors (CMOSs) are prevalent technology for capturing images, from digital astrophotography to machine vision inspection. Silicon based CCDs or CMOSs can be sensitive to a range of wavelengths as in human’s eye, but a litter wider. However, there is still a wide range of electromagnetic waves that cannot be sensed with CCD or CMOS based cameras. Thus, some MV systems expand the functions at infrared (IR), ultraviolet (UV), or X-ray wavelengths by using different photon-electrical materials or even thermal-electrical materials [1], [2]. Hence, infrared machine vision (IRMV) is an important supplement to MV. Devices included in a typical IRMV system are infrared cameras and lenses sensing the IR rays from 760 nm to 1 mm in wavelength, PCs for image processing and controllers. Excitation sources, instead, are optional and should be used depending on the actual application. Such distinction of thermal IRMV systems led to both passive thermography (PT) and active thermography (AT). PT can be used to obtain thermal images of an object that is constantly and naturally in disequilibrium from its environment without a controllable excitation source. AT, instead, is a well-established non-destructive testing (NDT) method that is used when internal defects need to be assessed in thermal-equilibria materials by exciting with an external thermal source to create a thermal gradient into the material.

It can be seen that both conventional MV based on visible light and IRMV based on IR rays are inseparable with machine learning (ML). In recent years, as for the rapid development of deep learning (DL), particularly with a class of ML algorithms that use multiple layer networks to progressively extract higher level features from the raw input, both MV and IRMV have obtained increasing interest in the fields of medicine, industry and architecture [3], [4], [5], [6]. The rapid development of deep learning makes IRMV more intelligent and automated thus enhancing its range of applications. After reviewing the related papers published in the recent 3–5 years, this work introduces firstly and systematically the principle of IRMV, cameras, excitation sources and data processes used in IRMV. In Section 2, DL models in IRMV are reviewed. Then, the applications of DL in passive TIRMV in different areas are presented and compared in Section 3. The application of DL in active TIRMV for thermography NDT is reported in Section 4. The DL based on various platforms such as unmanned aerial vehicles (UAV), mobile phones and embedded systems are discussed in Section 5. Section 6 forecasts the development trends of IRMV. Finally, conclusions are drawn in the last Section.

Section snippets

Principle, camera, sources and data of IRMV

Infrared-imaging based machine vision (IRMV) is the technology used to automatically inspect, detect, and analyse infrared images (or infrared videos) obtained by recording the intensity of IR light emitted or reflected by observed objects. An alternative definition of IRMV is the ability of machines to create images by infrared (IR) rays radiated or reflected by objects, where IR rays are electromagnetic waves with wavelengths ranging from 760 nm to 1 mm. Through the information extracted from

Deep learning in passive thermography

The usage of DL models including CNN, RNN, Autoencoder (AE), Restricted Boltzmann Machine (RBM) and Generative Adversarial Network (GAN) in PT are quite more frequent than those in AT. Typical cases are introduced and analysed according to the specific application.

DL in active thermography NDT

Like any other non-destructive testing methods, the objective of active thermography is to detect and evaluate the defects or discontinuity in materials. Currently, the usage of DL in active thermography is fewer than that in passive thermography. As active thermography NDT always generates the thermal video, we divide DL in active thermography NDT into three categories based on the structure of input data.

UAV-based IRT and DL

Frankly speaking, there are a lot of works on passive thermography with the help of UAV [92], [93], [94]. Here, we just want to talk the DL for the data collected by UAV carried thermal camera. In [95], the authors compare several methods for detecting failed solar panels, and found that drone detection is the most efficient. The thermographic images captured by drone are collected and various types of panel failures are explained. An examination by deep learning based on convolution neural

Trends

According to the above review, we can conclude the following trends:

  • (1)

    The small amount of data used for training results that the trained models are specific to a particular scenario and lack generalizability. In addition to waiting for the price of infrared cameras to decrease, researchers should give priority to the fields that urgently need infrared cameras, such as target recognition in special scenes and defect detection.

  • (2)

    In passive IRMV, DL of model fusion is an important topic in

Conclusions

Thermal infrared machine vision is classified into passive thermography and active thermography based on the fact whether a controllable excitation source is required. Passive thermography is important supplement for conventional MV based on visible light camera and active thermography is an important non-destructive testing method for quality and safety. The development of deep learning and UAV makes IRMV more intelligent and automated. This paper introduces firstly and systematically the

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The work was supported by National Natural Science Foundation of China under Grant No. 61811530331, 61901167, Royal Society Newton Mobility Grant, IEC\NSFC\170387, China Hunan Province Science&Technology department under Grant No. 2018RS3039, Changsha Science&Technology Bureau under Grant No. CSKJ2020-19, and Open Fundation from Laboratory of Science and Technology on Integrated Logistics Support, National University of Defense Technology under Grant No. 6142003200205. The authors are also

References (100)

  • R. Ali et al.

    Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer

    Constr. Build. Mater.

    (2019)
  • Y. Li

    Rotating machinery fault diagnosis based on convolutional neural network and infrared thermal imaging

    Chinese J. Aeronautics

    (2020)
  • M.W. Akram

    Automatic detection of photovoltaic module defects in infrared images with isolated and develop-model transfer deep learning

    Sol. Energy

    (2020)
  • C. Hu

    LSTM-RNN-based defect classification in honeycomb structures using infrared thermography

    Infrared Phys. Technol.

    (2019)
  • J. Xie

    Improving visibility of rear surface cracks during inductive thermography of metal plates using Autoencoder

    Infrared Phys. Technol.

    (2018)
  • N. Saeed

    Automatic defects detection in CFRP thermograms, using convolutional neural networks and transfer learning

    Infrared Phys. Technol.

    (2019)
  • Q. Luo

    Temporal and spatial deep learning network for infrared thermal defect detection

    NDT E Int.

    (2019)
  • R. Marani

    Modeling and classification of defects in CFRP laminates by thermal non-destructive testing

    Compos. B Eng.

    (2018)
  • T. D'Orazio

    Defect detection in aircraft composites by using a neural approach in the analysis of thermographic images

    NDTE Int.

    (2005)
  • A. Darabi et al.

    Neural network based defect detection and depth estimation in TNDE

    NDT E Int.

    (2002)
  • A. Buzmakov, et al., Overview of Machine Vision Methods in X-ray Imaging and Microtomography, in: A. Verikas, et al.,...
  • J. Lemley et al.

    Deep Learning for Consumer Devices and Services Pushing the limits for machine learning, artificial intelligence, and computer vision

    IEEE Consum. Electron. Mag.

    (2017)
  • N. Akhtar et al.

    Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey

    IEEE Access

    (2018)
  • A. Voulodimos

    Deep Learning for Computer Vision: A Brief Review

    Comput. Intell. Neurosci.

    (2018)
  • F. Khodayar et al.

    Infrared thermography and NDT: 2050 horizon

    Quantit. InfraRed Thermogr. J.

    (2016)
  • N. Dyakonova, et al., Terahertz vision using field effect transistors detectors arrays, in: 2018 22nd International...
  • A. Nadjib Danial, Using High Speed Shutter to Reduce Motion Blur in a Microbolometer,...
  • N. Rajic et al.

    A performance comparison between cooled and uncooled infrared detectors for thermoelastic stress analysis

    Quantit. InfraRed Thermogr. J.

    (2014)
  • B. Deng

    Line Scanning Thermography Reconstruction Algorithm for Defects Inspection with Novel Velocity Estimation and Image Registration

    IEEE Sens. J.

    (2020)
  • Z. Wang

    Comparative analysis of eddy current pulsed thermography and long pulse thermography for damage detection in metals and composites

    NDT E Int.

    (2019)
  • Z. He et al.

    Joint Scanning Laser Thermography Defect Detection Method for Carbon Fiber Reinforced Polymer

    IEEE Sens. J.

    (2020)
  • J. West et al.

    Spring research presentation: A theoretical foundation for inductive transfer

    Brigham Young Univ. College Phys. Math. Sci.

    (2007)
  • S.J. Pan et al.

    A survey on transfer learning

    IEEE Trans. Knowl. Data Eng.

    (2010)
  • W. Li, et al. Deep domain adaptive object detection: a survey, in: 2020 IEEE symposium series on computational...
  • W. Li, et al., Unsupervised Image-generation Enhanced Adaptation for Object Detection in Thermal images. arXiv preprint...
  • C. Herrmann, M. Ruf, J. Beyerer. CNN-based thermal infrared person detection by domain adaptation, in: Autonomous...
  • S.M. Iranmanesh

    Deep Cross Polarimetric Thermal-to-visible Face Recognition. arXiv

    Comput. Vision Pattern Recog.

    (2018)
  • H. Zhang

    Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks

    Int. J. Comput. Vision

    (2019)
  • C. Chen, A. Ross, Matching thermal to visible face images using a semantic-guided generative adversarial network, in:...
  • H. Zhang et al., Systematic infrared image quality improvement using deep learning based techniques. SPIE Remote...
  • Kwasniewska, A., et al., Super-resolved thermal imagery for high-accuracy facial areas detection and analysis....
  • F. Almasri et al.

    Multimodal Sensor Fusion in Single Thermal Image Super-Resolution

    (2019)
  • D. Xu

    Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

    Appl. Sci.

    (2020)
  • R. Hou

    VIF-Net: An Unsupervised Framework for Infrared and Visible Image Fusion

    IEEE Trans. Comput. Imag.

    (2020)
  • M.d.F.O. Baffa, L.G. Lattari, Convolutional Neural Networks for Static and Dynamic Breast Infrared Imaging...
  • S. Guan, N. Kamona, M. Loew, Segmentation of Thermal Breast Images Using Convolutional and Deconvolutional Neural...
  • S.T. Kakileti, G. Manjunath, H.J. Madhu, Cascaded CNN for View Independent Breast Segmentation in Thermal Images, in:...
  • I. Cruz-Vega

    Deep Learning Classification for Diabetic Foot Thermograms

    Sensors

    (2020)
  • H. Maldonado

    Automatic detection of risk zones in diabetic foot soles by processing thermographic images taken in an uncontrolled environment

    Infrared Phys. Technol.

    (2020)
  • F. Porikli, A. Yilmaz, Object Detection and Tracking,...
  • Cited by (106)

    View all citing articles on Scopus
    View full text