ReviewInfrared machine vision and infrared thermography with deep learning: A review
Introduction
Human eyes are sensitive to electromagnetic rays with wavelengths ranging from 400 to 760 nm (i.e. the visible light, VL). Machine vision (MV) is the technology that enables imaging machines to inspect, recognize and analyse images as human beings. A typical MV-based system is shown in Fig. 1(a), which includes VL cameras with lenses, light sources, personal computers (PCs, or embedded system) for image processing and actuators. Couple charged devices (CCDs) and complementary metal oxide semiconductors (CMOSs) are prevalent technology for capturing images, from digital astrophotography to machine vision inspection. Silicon based CCDs or CMOSs can be sensitive to a range of wavelengths as in human’s eye, but a litter wider. However, there is still a wide range of electromagnetic waves that cannot be sensed with CCD or CMOS based cameras. Thus, some MV systems expand the functions at infrared (IR), ultraviolet (UV), or X-ray wavelengths by using different photon-electrical materials or even thermal-electrical materials [1], [2]. Hence, infrared machine vision (IRMV) is an important supplement to MV. Devices included in a typical IRMV system are infrared cameras and lenses sensing the IR rays from 760 nm to 1 mm in wavelength, PCs for image processing and controllers. Excitation sources, instead, are optional and should be used depending on the actual application. Such distinction of thermal IRMV systems led to both passive thermography (PT) and active thermography (AT). PT can be used to obtain thermal images of an object that is constantly and naturally in disequilibrium from its environment without a controllable excitation source. AT, instead, is a well-established non-destructive testing (NDT) method that is used when internal defects need to be assessed in thermal-equilibria materials by exciting with an external thermal source to create a thermal gradient into the material.
It can be seen that both conventional MV based on visible light and IRMV based on IR rays are inseparable with machine learning (ML). In recent years, as for the rapid development of deep learning (DL), particularly with a class of ML algorithms that use multiple layer networks to progressively extract higher level features from the raw input, both MV and IRMV have obtained increasing interest in the fields of medicine, industry and architecture [3], [4], [5], [6]. The rapid development of deep learning makes IRMV more intelligent and automated thus enhancing its range of applications. After reviewing the related papers published in the recent 3–5 years, this work introduces firstly and systematically the principle of IRMV, cameras, excitation sources and data processes used in IRMV. In Section 2, DL models in IRMV are reviewed. Then, the applications of DL in passive TIRMV in different areas are presented and compared in Section 3. The application of DL in active TIRMV for thermography NDT is reported in Section 4. The DL based on various platforms such as unmanned aerial vehicles (UAV), mobile phones and embedded systems are discussed in Section 5. Section 6 forecasts the development trends of IRMV. Finally, conclusions are drawn in the last Section.
Section snippets
Principle, camera, sources and data of IRMV
Infrared-imaging based machine vision (IRMV) is the technology used to automatically inspect, detect, and analyse infrared images (or infrared videos) obtained by recording the intensity of IR light emitted or reflected by observed objects. An alternative definition of IRMV is the ability of machines to create images by infrared (IR) rays radiated or reflected by objects, where IR rays are electromagnetic waves with wavelengths ranging from 760 nm to 1 mm. Through the information extracted from
Deep learning in passive thermography
The usage of DL models including CNN, RNN, Autoencoder (AE), Restricted Boltzmann Machine (RBM) and Generative Adversarial Network (GAN) in PT are quite more frequent than those in AT. Typical cases are introduced and analysed according to the specific application.
DL in active thermography NDT
Like any other non-destructive testing methods, the objective of active thermography is to detect and evaluate the defects or discontinuity in materials. Currently, the usage of DL in active thermography is fewer than that in passive thermography. As active thermography NDT always generates the thermal video, we divide DL in active thermography NDT into three categories based on the structure of input data.
UAV-based IRT and DL
Frankly speaking, there are a lot of works on passive thermography with the help of UAV [92], [93], [94]. Here, we just want to talk the DL for the data collected by UAV carried thermal camera. In [95], the authors compare several methods for detecting failed solar panels, and found that drone detection is the most efficient. The thermographic images captured by drone are collected and various types of panel failures are explained. An examination by deep learning based on convolution neural
Trends
According to the above review, we can conclude the following trends:
- (1)
The small amount of data used for training results that the trained models are specific to a particular scenario and lack generalizability. In addition to waiting for the price of infrared cameras to decrease, researchers should give priority to the fields that urgently need infrared cameras, such as target recognition in special scenes and defect detection.
- (2)
In passive IRMV, DL of model fusion is an important topic in
Conclusions
Thermal infrared machine vision is classified into passive thermography and active thermography based on the fact whether a controllable excitation source is required. Passive thermography is important supplement for conventional MV based on visible light camera and active thermography is an important non-destructive testing method for quality and safety. The development of deep learning and UAV makes IRMV more intelligent and automated. This paper introduces firstly and systematically the
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The work was supported by National Natural Science Foundation of China under Grant No. 61811530331, 61901167, Royal Society Newton Mobility Grant, IEC\NSFC\170387, China Hunan Province Science&Technology department under Grant No. 2018RS3039, Changsha Science&Technology Bureau under Grant No. CSKJ2020-19, and Open Fundation from Laboratory of Science and Technology on Integrated Logistics Support, National University of Defense Technology under Grant No. 6142003200205. The authors are also
References (100)
Detection of potato tubers using an ultraviolet imaging-based machine vision system
Biosyst. Eng.
(2010)Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection
Constr. Build. Mater.
(2017)- et al.
Optically and Non-optically Excited Thermography for Composites: A review
Infrared Phys. Technol.
(2016) - et al.
Motion deblurring of infrared images from a microbolometer camera
Infrared Phys. Technol.
(2010) Imaging of barely visible impact damage on a composite panel using nonlinear wave modulation thermography
NDT E Int.
(2018)- et al.
Nonlinear air-coupled thermosonics for fatigue micro-damage detection and localisation
NDT E Int.
(2018) Volume or inside heating thermography using electromagnetic excitation for advanced composite materials
Int. J. Therm. Sci.
(2017)Image processing based quantitative damage evaluation in composites with long pulse thermography
NDT E Int.
(2018)Enhanced pre-processing of thermal data in long pulse thermography using the Levenberg-Marquardt algorithm
Infrared Phys. Technol.
(2019)- et al.
Deep visual domain adaptation: A survey
Neurocomputing
(2018)
Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer
Constr. Build. Mater.
Rotating machinery fault diagnosis based on convolutional neural network and infrared thermal imaging
Chinese J. Aeronautics
Automatic detection of photovoltaic module defects in infrared images with isolated and develop-model transfer deep learning
Sol. Energy
LSTM-RNN-based defect classification in honeycomb structures using infrared thermography
Infrared Phys. Technol.
Improving visibility of rear surface cracks during inductive thermography of metal plates using Autoencoder
Infrared Phys. Technol.
Automatic defects detection in CFRP thermograms, using convolutional neural networks and transfer learning
Infrared Phys. Technol.
Temporal and spatial deep learning network for infrared thermal defect detection
NDT E Int.
Modeling and classification of defects in CFRP laminates by thermal non-destructive testing
Compos. B Eng.
Defect detection in aircraft composites by using a neural approach in the analysis of thermographic images
NDTE Int.
Neural network based defect detection and depth estimation in TNDE
NDT E Int.
Deep Learning for Consumer Devices and Services Pushing the limits for machine learning, artificial intelligence, and computer vision
IEEE Consum. Electron. Mag.
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
IEEE Access
Deep Learning for Computer Vision: A Brief Review
Comput. Intell. Neurosci.
Infrared thermography and NDT: 2050 horizon
Quantit. InfraRed Thermogr. J.
A performance comparison between cooled and uncooled infrared detectors for thermoelastic stress analysis
Quantit. InfraRed Thermogr. J.
Line Scanning Thermography Reconstruction Algorithm for Defects Inspection with Novel Velocity Estimation and Image Registration
IEEE Sens. J.
Comparative analysis of eddy current pulsed thermography and long pulse thermography for damage detection in metals and composites
NDT E Int.
Joint Scanning Laser Thermography Defect Detection Method for Carbon Fiber Reinforced Polymer
IEEE Sens. J.
Spring research presentation: A theoretical foundation for inductive transfer
Brigham Young Univ. College Phys. Math. Sci.
A survey on transfer learning
IEEE Trans. Knowl. Data Eng.
Deep Cross Polarimetric Thermal-to-visible Face Recognition. arXiv
Comput. Vision Pattern Recog.
Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks
Int. J. Comput. Vision
Multimodal Sensor Fusion in Single Thermal Image Super-Resolution
Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network
Appl. Sci.
VIF-Net: An Unsupervised Framework for Infrared and Visible Image Fusion
IEEE Trans. Comput. Imag.
Deep Learning Classification for Diabetic Foot Thermograms
Sensors
Automatic detection of risk zones in diabetic foot soles by processing thermographic images taken in an uncontrolled environment
Infrared Phys. Technol.
Cited by (106)
Time-prior-based stacking ensemble deep learning model for ship infrared automatic target recognition in complex maritime scenarios
2024, Infrared Physics and TechnologyCrack detection of masonry structure based on thermal and visible image fusion and semantic segmentation
2024, Automation in ConstructionGlobal attention network with multiscale feature fusion for infrared small target detection
2024, Optics and Laser TechnologyMachine learning models applied to moisture assessment in building materials
2023, Construction and Building MaterialsA reliability study on automated defect assessment in optical pulsed thermography
2023, Infrared Physics and Technology