Temporal image analytics for abnormal construction activity identification
Introduction
For the construction industry, maintaining high performance in attempt to reduce project delay is a common goal [1]. Meeting project deadlines is the most critical aspect for the owners during construction [2]. Various circumstances may compromise productivity or even cause construction suspension; consequently, identification of abnormal activities in a timely fashion is vital for effective project control. To discover the anomaly, construction process analysis is indispensable. Typical approaches for this goal include on-site supervision and monitoring of surveillance videos, which are labor-intensive and time consuming.
With the booming popularity of the Internet of Things (IoT) and artificial intelligence, numerous studies have provided applicable frameworks to automate construction progress monitoring [[3], [4], [5], [6], [7]]. For example, Inertial Measurement Units (IMUs) were installed on heavy equipment to measure cycle times [8], depth sensors were applied to capture motion data, and the Support Vector Machine (SVM) and Hidden Markov Model (HMM) were adopted for worker action classifications [9]. Tang et al. [10] not only applied Faster Region-proposal Convolutional Neural Network (Faster R-CNN) combining Feature Pyramid Network (FPN) to detect construction resources, but further took interaction between workers and construction resource instances into consideration for safety improvement. In addition, Crew Balance Charts (CBCs) were used to visualize construction activities for field managers to obtain comprehensive information [11].
While existing studies have established automatic monitoring systems [[3], [4], [5], [6], [7], [8], [9], [10]], most of these works focus on the methodologies of construction resources detection, action and activity recognition, but detection of abnormal events. In other words, there are still opportunities to improve activity monitoring and analysis. This paper proposes an image-based analytics framework to automatically identify irregular activities through an alternative form of the CBC, called line chart. In this line chart, operation cycles which are irregular compared with standard operations are identified and highlighted. Meanwhile, a log is created with starting and ending times of each irregular event. Irregular operations are potential abnormal events that may decrease productivity and may be related to construction safety. With the provided information, field manager can directly acquire surveillance videos of irregular operations to investigate the cause of anomalies.
The remaining sections of the paper are summarized as follows. Section 2 reviews the related works. Section 3 states the pipeline of image analytics. Section 4 presents the quantitative results. Section 5 discusses the outcomes of image analysis, contribution and limitations of this study. Finally, Section 6 concludes the work.
Section snippets
Literature review
Progress monitoring on the construction jobsite is vital to the overall project delivery. Recognition of construction activities provides the opportunity for more detailed analysis of the process. Numerous studies adopted IoT sensors, such as IMUs or accelerometers, to collect essential action data [6,8,12]. However, installation of those devices on every worker and piece of equipment could be costly and inconvenient [13]. In addition, workers may feel uncomfortable or inconvenient to wear
Methodology
This study follows the typical pipeline consisting the following four steps: object detection, object tracking, action recognition, and operational analysis (Fig. 1). The objects of interest, such as workers, excavators and dump trucks, are detected in images. Second, detection results are associated in attempt to distinguish individual trajectories. In other words, tracking is conducted. Third, the time-series data obtained from the previous module are used to recognize particular actions.
Experimental results
Although several studies regarding automatic construction monitoring were conducted, little data has been provided for public usage. The earthmoving operation dataset, with excavators and dump trucks, provided by Roberts and Golparvar-Fard [5] is available, and this section exclusively focuses on these activities.
Discussion
In this section, insights regarding the analysis are provided, and the contribution and limitations of the work are discussed.
Conclusion
This study proposes an image analytics approach for field managers to examine irregular activities which may detain the schedule. The framework consists of four modules: object detection, object tracking, action recognition, and operational analysis.
For object detection, the pre-trained Faster R-CNN was adopted to recognize objects in the image taken from construction sites. The model goes through transfer learning with the ACID for better detection results.
For object tracking, both SORT and
Declaration of Competing Interest
We declare no conflict of interest to any individual.
Acknowledgment
The authors would like to thank the Research Center for Building Information Modeling and Management at National Taiwan University for the support that has made this work possible. The second author would also like to thank the Ministry of Science and Technology (MOST) of Taiwan for the support of grant MOST 109-2221-E-002-015.
References (74)
- et al.
Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles
Autom. Constr.
(2019) - et al.
Automated excavators activity recognition and productivity analysis from construction site surveillance videos
Autom. Constr.
(2020) - et al.
End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level
Autom. Constr.
(2019) - et al.
Construction activity recognition with convolutional recurrent networks
Autom. Constr.
(2020) - et al.
Vision-based workface assessment using depth images for activity analysis of interior construction operations
Autom. Constr.
(2014) - et al.
Application of dynamic time warping to the recognition of mixed equipment activities in cycle time measurement
Autom. Constr.
(2018) - et al.
Human-object interaction recognition for automatic construction site safety inspection
Autom. Constr.
(2020) - et al.
Semi-supervised near-miss fall detection for ironworkers with a wearable inertial measurement unit
Autom. Constr.
(2016) - et al.
Vision-based detection and visualization of dynamic workspaces
Autom. Constr.
(2019) - et al.
Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks
Autom. Constr.
(2018)
Convolutional neural networks: computer vision-based workforce activity assessment in construction
Autom. Constr.
A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory
Autom. Constr.
Two-step long short-term memory method for identifying construction activities through positional and attentional cues
Autom. Constr.
Detecting non-hardhat-use by a deep learning method from far-field surveillance videos
Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach
Advanced Engineering Informatics
Temporal dynamic appearance modeling for online multi-person tracking
Comput. Vis. Image Underst.
Tracking multiple construction workers through deep learning and the gradient based method with re-matching based on multi-object tracking accuracy
Autom. Constr.
A multivariate kolmogorov-smirnov test of goodness of fit
Statistics & Probability Letters
Activity analysis for direct-work rate improvement in construction
J. Constr. Eng. Manag.
Visual data and predictive analytics for proactive project controls on construction sites
Automated worker activity analysis in indoor environments for direct-work rate improvement from long sequences of rgb-d images
Crowdsourcing construction activity analysis from jobsite video streams
J. Constr. Eng. Manag.
Part based model and spatial–temporal reasoning to recognize hydraulic excavators in construction images and videos
Autom. Constr.
Focal loss for dense object detection
Object detection from video tubelets with convolutional neural networks
Vision-based construction activity analysis in long video sequences via hidden markov models: Experiments on earthmoving operations
Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and Imagenet?
Deep residual learning for image recognition
Vision-based construction worker activity analysis informed by body posture
J. Comput. Civ. Eng.
Yolov3: An incremental improvement
Quo vadis, action recognition? a new model and the kinetics dataset
Stacked hourglass networks for human pose estimation
Computer vision-based video interpretation model for automated productivity analysis of construction operations
J. Comput. Civ. Eng.
Outlier Detection
Outlier Detection with Globally Optimal Exemplar-Based GMM, in: Proceedings of the 2009 SIAM International Conference on Data Mining
Kernel density outlier detector
Deep learning for anomaly detection: A survey
Cited by (27)
Contextual multimodal approach for recognizing concurrent activities of equipment in tunnel construction projects
2024, Automation in ConstructionVision-based excavator pose estimation for automatic control
2024, Automation in ConstructionAutomated vision-based construction progress monitoring in built environment through digital twin
2023, Developments in the Built EnvironmentVideo surveillance using deep transfer learning and deep domain adaptation: Towards better generalization
2023, Engineering Applications of Artificial IntelligenceDigital technology for quality management in construction: A review and future research directions
2022, Developments in the Built EnvironmentCitation Excerpt :CV has been used for construction activities identification (Seo et al., 2015), site productivity assessment (Luo et al., 2018), detection and visualization of dynamic workspaces (Luo et al., 2019), and real-time structural health (Dong and Catbas, 2021). Visual data collected from construction sites are used for site safety (Fang et al., 2020b), abnormal construction activity identification (Lin et al., 2021), and automated postconstruction quality assessment for defects in inspection (Liu et al., 2017). Blockchain is defined as a distributed ledger technology characterized by decentralized operations across a consensus mechanism network (i.e., peer to peer), where all data are stored as blocks that are immutable once joined and authenticated in a chain (Viriyasitavat et al., 2019).
Real-time indoor localization with visual SLAM for in-building emergency response
2022, Automation in Construction