A vision-based method for automatic tracking of construction machines at nighttime based on deep learning illumination enhancement

doi:10.1016/j.autcon.2021.103721

Automation in Construction

Volume 127, July 2021, 103721

https://doi.org/10.1016/j.autcon.2021.103721 Get rights and content

Highlights

•
A vision-based method is proposed for tracking construction machines at nighttime.
•
Deep learning illumination enhancement is integrated to overcome low lighting issues.
•
The results of nine nighttime videos achieved 95.1% in MOTA and 75.9% in MOTP.
•
Illumination enhancement module improved tracking robustness by 41% in extreme conditions.

Abstract

Nighttime construction has been widely conducted in many construction scenarios, but it is also much riskier due to low lighting conditions and fatiguing environments. Therefore, this study proposes a vision-based method specifically for automatic tracking of construction machines at nighttime by integrating the deep learning illumination enhancement. Five main modules are involved in the proposed method, including illumination enhancement, machine detection, Kalman filter tracking, machine association, and linear assignment. Then, a testing experiment based on nine nighttime videos is conducted to evaluate the tracking performance using this approach. The results show that the method developed in this study achieved 95.1% in MOTA and 75.9% in MTOP. Compared with the baseline method SORT, the proposed method has improved the tracking robustness of 21.7% in nighttime construction scenarios. The proposed methodology can also be used to help accomplish automated surveillance tasks in nighttime construction to improve the productivity and safety performance.

Introduction

Construction at nighttime is a common practice in many construction scenarios including pavement maintenance, highway construction, and railway construction [1]. Compared with daytime construction, nighttime construction has some obvious advantages [2]. For instance, operating outdoor construction activities (e.g., for the pavement project) at nighttime can reduce the interferences with traveling vehicles to avoid traffic congestions. Furthermore, the working environment of nighttime construction is less complex than that during the daytime (e.g., fewer pedestrians and traffics), which can improve the crew productivity to some extent. Especially, in the summer months, the temperature at nighttime is much lower than that at daytime. The cooler construction condition could speed up material delivery cycles and shorten machinery idling time.

In addition, awareness of construction safety at nighttime has been a major concern in the industry since nighttime construction naturally creates hazardous working conditions. The results from the study of Arditi et al. [3] indicated that the safety risks at nighttime can be five times higher than daytime construction due to some significant factors: a) the lower illumination conditions; and b) workers and machine operators are prone to be fatigue. By analyzing 20,997 construction industry accident records in the United States from 1997 to 2012, Kang et al. [4] identified 22.8% of construction accidents belonged to struck-by accidents, which is the second most influential factor for construction safety. Therefore, monitoring construction machines at nighttime is beneficial to avoid struck-by hazards between works and machines, which is important for facilitating a relatively safe working environment. Currently, the manual method still acts as the primary approach for monitoring nighttime construction, but its usage is error-prone and time-consuming. As such, automatically tracking construction machines with retrieving the trajectory information can be treated as an alternative way to help identify the potential collisions and reduce the risks of struck-by accidents in nighttime construction [5].

Recently, tracking construction machines by using vision-based methods has become a promising way considering cameras are less costly and simply to install [6]. Extensive research studies have been conducted on tracking construction machines from videos for evaluating crew productivity and enhancing site safety in the area of construction engineering and management. For instance, Chen et al. [7] automatically calculated the productivity of excavators in earthmoving works by tracking excavators using vision-based methods. Kim et al. [8] proposed an automatic method for analyzing the productivity of tunnel earthmoving by means of tracking excavators and dump trucks. Yan et al. [9] adopted a struck-by accident monitoring method based on tracking heavy machines and workers from videos in order to enhance site safety.

It can be seen that most of the existing vision-based methods for tracking construction machines tend to focus on daytime construction [10], but it is difficult to achieve reliable performance when using these approaches in nighttime scenarios. Tracking construction machines from nighttime videos is challenging due to the limited visibility and significant illumination variations [11]. In nighttime construction, the lighting conditions are usually poor and construction machines are prone to be partially invisible. Meanwhile, the illumination variations can easily create blurs of construction objects, which will trigger some difficulties in distinguishing one construction machine from others through tracking. Therefore, it is urgent and necessary to develop a more robust appearance model for automatically tracking construction machines in nighttime scenarios.

The main objective of this research is to propose a vision-based method for automatic tracking of construction machines at nighttime. Specifically, this approach can enhance the image quality for tracking in low lighting conditions and explore the robust appearance model for describing machine objects when facing illumination variations. The proposed tracking method is expected to produce stable and precise bounding boxes of construction machines from nighttime videos in order to generate trajectory information. By integrating the method developed in this study, many surveillance tasks in nighttime construction can be accomplished automatedly including crew productivity evaluation, machine idling analysis, and collision alerts, which will make nighttime construction more efficient and safer.

Section snippets

Literature review

Automatic tracking of construction entities by vision-based methods is an important topic in the research community, which is useful for project monitoring, materials transportation, and site safety. In this section, the literature regarding the vision-based tracking methods and their applications in construction will be reviewed. Since this research tries to integrate illumination enhancement methods for nighttime tracking, the development of the image illumination enhancement will also be

Methodology

In this section, the overall framework of the proposed methodology is introduced firstly. Then, five main modules involved in this vision-based method are described in detail, that is, illumination enhancement, machine detection, Kalman filter tracking, machine association, and linear assignment.

Experiments and results

The experiments and results are presented in this section. First, the implementation of the proposed method is introduced. Then, the experimental setup, evaluation metrics, and experimental results are reported in details.

Discussion

In this section, the experimental results have been analyzed in depth. The discussion includes the influences of the illumination enhancement module and CNN feature extractor. Moreover, the proposed method has been tested on daytime videos to validate its feasibility. The influences of training images dataset have also been discussed.

Conclusions and future works

This research presents a vision-based method for automatic tracking of construction machines in nighttime videos. Specifically, the proposed method adopts the deep learning illumination enhancement method to improve the lighting conditions in nighttime videos and a CNN feature extractor as the appearance model to distinguish construction machines. After applying this approach to nine nighttime testing videos, the tracking performance of construction machines was relatively satisfactory. For

Declaration of Competing Interest

No potential conflict of interest was reported by the authors.

Acknowledgement

The authors would like to thank Professor Shih-Chung Kang from the University of Alberta for providing us the ACID dataset for training. More information about the ACID dataset can be found from the link: www.acidb.ca. This work was supported by the National Natural Science Foundation of China under Grant (number 72002152).

References (64)

J. Yang et al.
Construction performance monitoring via still images, time-lapse photos, and video streams: now, tomorrow, and the future
Adv Eng Inform
(2015)
B. Shao et al.
Fatal accident patterns of building construction activities in China
Saf Sci
(2019)
D. Arditi et al.
Fatal accidents in nighttime vs. daytime highway construction work zones
J Saf Res
(2007)
C. Chen et al.
Automated excavators activity recognition and productivity analysis from construction site surveillance videos
Autom Constr
(2020)
H. Kim et al.
Analyzing context and productivity of tunnel earthmoving processes using imaging and simulation
Autom Constr
(2018)
D. Yuan et al.
Visual object tracking with adaptive structural convolutional network
Knowledge-Based Syst
(2020)
G. Ciaparrone et al.
Deep learning in video multi-object tracking: a survey
Neurocomputing.
(2020)
Z. Sun et al.
A survey of multiple pedestrian tracking based on tracking-by-detection framework
IEEE Trans Circuits Syst Video Technol
(2020)
O. Angah et al.
Tracking multiple construction workers through deep learning and the gradient based method with re-matching based on multi-object tracking accuracy
Autom Constr
(2020)
E. Konstantinou et al.
Adaptive computer vision-based 2D tracking of workers in complex environments
Autom Constr
(2019)

W. Zhou et al.

Construction safety and digital design: a review

Autom Constr

(2012)

M. Golparvar-Fard et al.

Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers

Adv Eng Inform

(2013)

D. Roberts et al.

End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level

Autom Constr

(2019)

H. Kim et al.

Vision-based object-centric safety assessment using fuzzy inference: monitoring struck-by accidents with moving objects

J Comput Civ Eng

(2016)

K.G. Lore et al.

LLNet: a deep autoencoder approach to natural low-light image enhancement

Pattern Recogn

(2017)

X. Yin et al.

A deep learning-based framework for an automated defect detection system for sewer pipes

Autom Constr

(2020)

A. Xuehui et al.

Dataset and benchmark for detecting moving objects in construction sites

Autom Constr

(2021)

H. Tajeen et al.

Image dataset development for measuring construction equipment recognition performance

Autom Constr

(2014)

Y. Kang et al.

Trends of fall accidents in the U.S. construction industry

J Constr Eng Manag

(2017)

B. Sherafat et al.

Automated methods for activity recognition of construction workers and equipment: state-of-the-art review

J Constr Eng Manag

(2020)

S. Xu et al.

Computer vision techniques in construction: a critical review

Arch Comput Methods Eng

(2020)

X. Yan et al.

Computer vision-based recognition of 3D relationship between construction entities for monitoring struck-by accidents

Comput Civ Infrastruct Eng

(2020)

P. Dendorfer et al.

MOT20: A Benchmark for Multi Object Tracking in Crowded Scenes

B. Xiao et al.

Two-dimensional visual tracking in construction scenarios: a comparative study

J Comput Civ Eng

(2018)

N. Wang et al.

Understanding and diagnosing visual tracking systems

J. Hu et al.

A comparative study of sampling analysis in the scene classification of optical high-spatial resolution remote sensing imagery

Remote Sens

(2015)

C. Bailer et al.

A superior tracking approach: Building a strong tracker through fusion

N. Dalal et al.

Histograms of oriented gradients for human detection

R. Lienhart et al.

An extended set of Haar-like features for rapid object detection

G. Gualdi et al.

Contextual information and covariance descriptors for people surveillance: an application for safety of construction workers

EURASIP J Image Video Process

(2011)

P. Voigtlaender et al.

Siam R-CNN: Visual tracking by re-detection

J.S. Supancic et al.

Self-paced learning for long-term tracking

Cited by (30)

An unsupervised low-light image enhancement method for improving V-SLAM localization in uneven low-light construction sites
2024, Automation in Construction
Construction robots have been increasingly adopted in construction projects to improve productivity, reduce risk, and speed up work cycles. Visual Simultaneous Localization and Mapping (V-SLAM) technology is widely used in construction robots due to its lightweight weight, cost-effectiveness, and provision of semantic information. On real construction sites, lighting conditions are often challenging due to factors such as uneven lighting intensity, inadequate exposure, or robots in backlit positions (e.g. roughcast houses without artificial lighting, underground car parks), which cause images captured under these conditions to exhibit low signal-to-noise ratio, uneven lighting, color distortion, noise, and other problems. These challenges make the extraction and matching of feature points in V-SLAM difficult, resulting in significant positioning errors or the inability to generate positioning trajectories for construction robotics. Although existing methods for enhancing low-light images can certainly improve image brightness, they still cannot effectively address the localization challenges brought about by low-light construction scenes with uneven illumination and noise. To address the interference of uneven and changing lighting conditions on V-SLAM localization in construction sites, this paper proposes the Unsupervised Reflectance Retinex and Noise model (URRN-Net) to enhance low-light construction images. By using a CNN model based on the Retinex theory to decompose the illumination characteristics of images, we achieve effective brightness restoration of uneven low-light images by utilizing unsupervised loss functions. URRE-Net can reduce the root mean square localization error by >65% compared to low-light images in the ORB-SLAM3 method, and the maximum error is reduced by >71% in on-site experiments. The proposed URRE-Net can be integrated with existing V-SLAM algorithms to provide more robust localization services for low-light construction site applications such as building operations (e.g., interior wall spraying robotic) or construction management tasks (e.g., automatic tunnel inspection).
Computer vision-based safety risk computing and visualization on construction sites
2023, Automation in Construction
Despite advancements in computer vision technology for construction site safety, the identification and evaluation of potential safety risks stemming from on-site hazardous objects and their spatial interactions is under-researched. Moreover, the subsequent visualization of risks, essential for effective safety management, remains insufficiently explored. This paper presents a model combining computer vision and the TOPSIS method to automate safety risk quantification and visualization during the construction process. Uniquely, it enhances risk predictions by dynamically monitoring the quantity and distance changes of hazard sources and their risk-related objects in real-time, beyond pre-construction risk identification and predictions. Experimental validation shows the model's effectiveness in risk quantification and visualization, with a high consistency ratio of 95% compared to expert manual assessments. This model lays the groundwork for more precise risk calculation and evaluation, fostering improved safety management decisions.
Hybrid DNN training using both synthetic and real construction images to overcome training data shortage
2023, Automation in Construction
Although deep neural network (DNN)-powered visual scene understanding is a driving factor in a transition toward construction digitalization and robotic automation, a shortage of construction training images has been a roadblock to achieving DNNs' maximum performance potential. This data shortage becomes more problematic in digitally monitoring field workers who perform a variety of activities in an unstructured outdoor construction environment. To address this issue, the authors present a construction worker-centric image synthetization approach that can automatically synthesize and label limitless artificial human images with diverse poses, activities, and outdoor imaging conditions. Using synthesized construction worker-centric images, the authors conduct training experiments to characterize the effects of synthetic images on DNN-powered worker detection. In addition, the authors explore the hybrid effects of synthetic and real images on DNN performance. Results showed that a synthetic image-trained model potentially performs well in diverse field conditions and can even detect construction workers who are missed by a real image-trained model. It was also shown that a hybrid use of synthetic and real images can reduce the number of necessary real training images by 50% and improve DNN performance by 16% on average, compared to when only one of the two data sources are adopted. Moreover, the data hybridity enabled DNNs to reach its near-maximum performance while scaling down the size of a real training dataset by up to 80%. These findings indicate that synthetic images have promising potential for worker-centric DNN training in that they enable higher performance while reducing the human effort needed for real construction image collection and labeling. This capability can help to address the problem of data shortage in construction and enable the training of more accurate and scalable DNN models. Furthermore, this will stimulate the development and implementation of visual artificial intelligence for robotic automation and digitization.
Mutually coupled detection and tracking of trucks for monitoring construction material arrival delays
2022, Automation in Construction
Citation Excerpt :
Several automatic multi-object tracking (MOT) methods can be adopted to achieve continuous and timely material arrival delays monitoring, such as the radio frequency identification (RFID) [7], the ultrawideband (UWB) [8], and the computer vision-based (CVB) methods [9,10]. Compared with the two former sensor-based methods, the CVB method has the advantages of non-invasiveness and easier access to informative data, and it has achieved significant progress in construction with the development of deep learning [11–15]. However, few studies have focused on CVB multiple truck tracking for monitoring construction material arrival delays, especially in PBC.
Construction project is sensitive to material arrival delays, which can cause schedule delays and budget overruns. The prompt detection of construction material arrival delays is necessary to recover disrupted projects in time. This paper explores computer vision-based (CVB) mutually coupled detection and tracking of transport trucks for monitoring construction material arrival delays. Through the mutually coupled mechanism, the truck detection and tracking algorithms complement each other in reducing accumulated tracking errors, modifying false positives and false negatives in detection, and stabilizing the detected bounding boxes. The experimental results indicate integrating a deep Convolutional Neural Network (CNN), a Kanade-Lucas-Tomasi (KLT) corner feature tracker, and a hash-based occlusion handling strategy can achieve high tracking precision compared to state-of-the-art trackers in construction. The field application results indicate that the proposed method can achieve the automatic monitoring of construction material arrival delays.
Vision-based method for tracking workers by integrating deep learning instance segmentation in off-site construction
2022, Automation in Construction
Citation Excerpt :
In the present research, a novel instance association method is proposed by integrating the Kalman filtering prediction and MIoU. Following the strategy put forward in our previous study [52], if a segmentation mask (maskis) is successfully matched with a tracking mask (maskit) and MIoU > 0.5, maskis will inherit the tracking ID of maskit. Meanwhile, pis will be used to update the corresponding Kalman Filtering tracker and maskis will replace maskit for the association at the next frame.
Vision-based tracking of construction workers is the fundamental step for many automated applications in off-site construction. This research proposes a vision-based method for tracking workers in off-site construction by integrating deep learning instance segmentation. The proposed method consists of three main modules including instance segmentation, instance association, and instance assignment. The instance segmentation module applies the Mask R-CNN algorithm to extract masks and bounding boxes of workers from videos. Then, an association matrix is constructed at each two consecutive frames based on the mask intersection-over-union and Kalman filtering. Finally, the instance assignment module solves the association matrix to produce tracking results. In experiments, the proposed method achieved the multiple object tracking accuracy of 96.4% and multiple object tracking precision of 86.2%. The testing results indicate the developed method can successfully track multiple workers when facing the challenges of occlusions and scale variations, etc.
Automated site-specific assessment of steel structures through integrating machine learning and fracture mechanics
2022, Automation in Construction
Despite the recent advances in using machine learning for crack identification of structures, the usefulness of artificial intelligence in guiding repair strategies is still limited due to the lack of information on the mechanism of crack formation and propagation. In this context, this paper develops an automated workflow to assess the fracture mechanism of steel structures directly from inspection images. Firstly, a U-Net is developed to find the pixel-location of a crack. Then, a Gaussian Process surrogate model is trained to predict stress intensity factors, K (indicator for fracture mechanism), for cracks with various dimensions using data from finite element simulations. In the end, by taking the dimensions of identified cracks as input, the trained Gaussian Process is used to estimate the K factors. The proposed workflow empowers an inspector to easily identify steel cracks, predict their propagation from a collection of raw images, and make rapid repair decisions on-site.

View all citing articles on Scopus

View full text

A vision-based method for automatic tracking of construction machines at nighttime based on deep learning illumination enhancement

Highlights

Abstract

Introduction

Section snippets

Literature review

Methodology

Experiments and results

Discussion

Conclusions and future works

Declaration of Competing Interest

Acknowledgement

Adv Eng Inform

Saf Sci

J Saf Res

Autom Constr

Autom Constr

Knowledge-Based Syst

Neurocomputing.

IEEE Trans Circuits Syst Video Technol

Autom Constr

Autom Constr

Autom Constr

Adv Eng Inform

Autom Constr

J Comput Civ Eng

Pattern Recogn

Autom Constr

Autom Constr

Autom Constr

Trends of fall accidents in the U.S. construction industry

J Constr Eng Manag

Automated methods for activity recognition of construction workers and equipment: state-of-the-art review

J Constr Eng Manag

Computer vision techniques in construction: a critical review

Arch Comput Methods Eng

Computer vision-based recognition of 3D relationship between construction entities for monitoring struck-by accidents

Comput Civ Infrastruct Eng

MOT20: A Benchmark for Multi Object Tracking in Crowded Scenes

Two-dimensional visual tracking in construction scenarios: a comparative study

J Comput Civ Eng

Understanding and diagnosing visual tracking systems

A comparative study of sampling analysis in the scene classification of optical high-spatial resolution remote sensing imagery

Remote Sens

A superior tracking approach: Building a strong tracker through fusion

Histograms of oriented gradients for human detection

An extended set of Haar-like features for rapid object detection

Contextual information and covariance descriptors for people surveillance: an application for safety of construction workers

EURASIP J Image Video Process

Siam R-CNN: Visual tracking by re-detection

Self-paced learning for long-term tracking