Artificial intelligence-empowered pipeline for image-based inspection of concrete structures

Highlights

• Deep learning is applied for image-based inspection of concrete structures.

• An artificial intelligence-empowered inspection pipeline is established.

• Anomaly detection and extraction reduce the enormous search space of defects.

• Defect classification algorithms are robust to various defect conditions.

Abstract

Inspection of civil infrastructure is a major challenge to engineers due to the limitations in existing practice, which are as laborious, time-consuming and prone to error. To address these issues, we have applied deep learning for image-based inspection of concrete defects of civil infrastructure, and have established an artificial intelligence-empowered inspection pipeline methodology. This innovative approach comprises anomaly detection, anomaly extraction and defect classification. The anomaly detection and extraction are used to identify defect regions from the enormous volume of image datasets, which used to be the common challenges encountered in automated visual inspections. The search space of defects is substantially reduced, i.e., at least 60% of the original volume of image datasets, with an average hit rate of ~88.7% and an average false alarm rate of ~14.2%. Following that, deep learning-based classifiers are used to categorize defects into appropriate classes. The assessment results show that the proposed inspection pipeline exhibits great capability in detecting, extracting and classifying defects subjected to various environmental and operational conditions, including lighting condition, camera distance and capturing angle, with an average testing accuracy of 95.6%.

Introduction

Hong Kong has a small land area of 1106 km² and extensive civil infrastructure–2101 km of roads, 747 km of railway networks, 15 major road tunnels, 1340 flyovers and bridges, and extensive sewage and water treatment works [1]. Such vast infrastructure is subjected to inevitable ageing and deterioration, which cause potential failure and poor compliance with environmental regulations. According to the Hong Kong 2019–2020 Budget, the territory's infrastructure requires at least HKD 4434 million (equivalent to USD 566 million) for maintenance expenses of this fiscal year. Inspection programs are regularly conducted to assess the health status of infrastructure, evaluate suspected risks and hazards, and plan necessary repair, replacement or rehabilitation. Over the years, manual visual inspection has been the most widely used approach, assisted by simple visual technologies. While experienced inspectors might be able to intuit problems from their observations, this conventional practice is time consuming and labor intensive, especially when the site is large and difficult to access. Furthermore, the inspection results are often prone to error due to subjective decisions inherited from the experience and knowledge of inspectors.

Visual sensing technologies (e.g., commercial off-the-shelf cameras) and computer vision-based methods have been intensively applied as alternatives to address the limitations of manual visual inspection. Most of these approaches made use of the common attributes of defects as features for detection. For instance, cracks were presumed to be narrow and elongated in shape, dark in appearance, and in distinct contrast to surrounding healthy concrete surfaces. Among different image processing techniques, edge detection is the most commonly used method for defect detection. It is accomplished by either applying convolution with specifically designed kernels/filter arrays on the spatial domain (e.g., Sobel, Canny, Gabor, and other line-emphasis filters) or masking specific frequency regions on the frequency domain (e.g., fast Haar transform and fast Fourier transform). Given its ability to identify cracks from images, edge detection has been applied to different infrastructure, such as bridges [2], asphalt pavements [3], general concrete structures [4,5], and concrete specimens in laboratory tests [6,7], among others. In addition to edge detection, Yamaguchi and his research team [8,9] developed percolation models that considered the connectivity of cracks among neighborhoods in order to detect cracks from images; Noh et al. [10] applied fuzzy C-means clustering to images, followed by choosing the cluster with the lowest average brightness value as the potential cracks. Following defect detection, segmentation is normally implemented with thresholds, which are manually determined or based on relevant statistical parameters, to extract the defects.

Although the above methods are simple and direct, inherent limitations, which hinder deployment in real applications, remain. Most of the time, multiple pre- and post-processing steps are inevitably performed to (i) enhance the features of defects by applying background subtraction and image smoothing and (ii) remove noises by conducting morphological operations (e.g., dilation, erosion, opening and closing), and operations on connected components based on the object attributes (e.g., perimeter, area and aspect ratio). Trial-and-error iterations are required to select the optimal combination of pre- and post-processing steps and the defect detection algorithms. However, this procedure is time consuming, cumbersome and labor intensive.

Machine learning algorithms have been used to improve inspection efficiency. In this context, such algorithms are used as classifiers to identify defects from images, based on extracted features. For instance, O'Byrne et al. [11] constructed a feature vector with six statistics of textural features and four pieces of information derived from the Gray Level Co-occurrence Matrix. They then used a support vector machine (SVM) classifier to identify the defect regions. Lattanzi and Miller [12] proposed a modified k-means clustering method to facilitate the extraction of shape and textural features and then examined the performance of a naïve Bayesian statistical classifier, a boosted J48 decision tree using adaptive boosting and a KNN instance-based classifier. Prasanna et al. [13] studied the performance of SVM, Adaboost and random forest in classifying defects, based on spatially tuned multiple features. However, most of these existing works heavily depend on feature engineering, in which certain rules are fixed for classifying subsets of regions in images according to handcrafted features. To some extent, defects in the civil infrastructure inevitably exist in complex variations against noisy and random backgrounds. Each inspected site is highly case-specific; that is, fine-tuning of previously established rules is required, thereby limiting the generic application of the aforementioned algorithms. Thus, robust methods that can be adapted to as many civil infrastructure as possible are urgently needed, to considerably boost the maintenance efficiency.

The recent promising achievement of deep learning in AI and computer vision provides unprecedented opportunities to overcome this challenge. Deep neural networks transform raw data into representative feature spaces via different combinations of linear and non-linear operations [14,15]. The transformed feature spaces can then be further transformed in accordance with the purpose of the tasks (e.g., categorical labels for image classification, prediction maps for semantic segmentation and bounding boxes for object detection). Such transformation is differentiable; thus, the gradient-descent type of optimization techniques can be used to update and optimize the learnable network parameters. These frameworks holistically integrate multiple processing stages of computer vision-based methods, including noise reduction, contrast enhancement, threshold selection, and feature extraction. Such deep learning techniques were mostly applied on crack detection for different civil infrastructure, such as railways [16], roads [17], nuclear power plants [18] and general concrete structures [[19], [20], [21], [22], [23], [24], [25]]. A comprehensive study has also been conducted by Dorafshan et al. [21] to assert the performance of deep learning models over common edge detection techniques in concrete crack detection. Limited research has been carried out for multiclass defect classification using deep learning techniques; most of the existing works focused on defect detection on sewer pipes [[26], [27], [28], [29]] and stadiums [30] and other general scenarios [31]. In addition, most previous works generally focused on controlled contexts, i.e., man-made conditions, which may not be ideal and could not be generalized in an actual inspection [12,13,32,33]. Similar deep learning-based surface-defect detection and classification in the industrial inspection have been reported [[34], [35], [36], [37], [38]]. However, these surface-defects are also mostly found under regular and controlled environment, unlike defects at civil infrastructure, which possibly exist in a wide range of dimensions, patterns and scales under different backgrounds. Hence, a classifier that can categorize as many types of defects as possible under various real and generalized environmental and operational conditions is urgently needed.

Furthermore, the vast deployment of low-cost, high-performance and handy vision sensors to achieve automated visual inspection typically generates massive image datasets. However, the defects cover only small areas of the entire civil infrastructure. In other words, this approach creates large search spaces of sparse yet critical defects, resulting in a wasteful consumption of the very time that is meant to be saved through automation. For instance, Yeum and his research team [32,39] adopted a structure-from motion algorithm to extract regions of interest on a highway sign truss with welded connections; this requires a specific signature at the observed structure, which is not available at most of the civil infrastructure. Hüthwohl and Brilakis [40] used a convolutional neural network to detect and disregard healthy concrete surfaces, so that more effort could be placed on assessing potentially unhealthy surface regions; however, the environmental conditions of civil infrastructure are generally complex, where various types of objects, e.g., machinery, pipes, cables and light sources, are found. All these objects are unavoidably captured as part of the images during inspection, which notoriously hinder subsequent image processing in retrieving and identifying relevant information on any potential defects. Specifically, this may result in a high likelihood of false positives (misclassifying non-relevant objects as defects) and reducing the trustworthiness and efficiency of the inspection methods [32]. To resolve all these problems, an artificial intelligence (AI)-empowered inspection pipeline is established; this is the objective of our study. Prior to the classification of defects, an anomaly detection technique is built to carry out an unsupervised screening to reliably reduce the search space of the defects. In short, this AI-empowered inspection pipeline is intended to streamline the visual inspection of the civil infrastructure, in terms of augmenting the degree of automation, as well as speeding up the processing and analyzing processes, while achieving satisfactory accuracy in defect detection and classification. As the initial step in developing this innovative pipeline, the focus is placed on detecting and classifying defects on concrete structures.

The organization of the paper is as follows. Image datasets used in this study are briefly described. Then, details of the proposed AI-empowered inspection pipeline, made up of anomaly detection, anomaly extraction and defect classification, are delineated. Assessment was conducted based on four different testing datasets and an open-source dataset, in order to validate the applicability of the proposed inspection pipeline.