Temporal image analytics for abnormal construction activity identification

https://doi.org/10.1016/j.autcon.2021.103572Get rights and content

Highlights

  • This is the first work that attempts to identify irregular construction activities from image based data.

  • Statistical approaches, such as the Chebyshev's theorem and Box Plot, were utilized for irregular activity identification.

  • This work utilized the line chart to color-code and visualize identified irregular activities.

  • State-of-the-art visual sensing models were utilized for object detection, tracking, and activity identification.

  • A customized tracking methodology was employed for better results.

Abstract

Abnormal activities on construction jobsites may compromise productivity and pose threat to workers' safety. This paper proposes the analysis of consecutive image sequences for automatic identification of irregular operations and their visualization. The data analytics is composed of four steps: object detection, object tracking, action recognition, and operational analysis. The Faster Region-proposal Convolutional Neural Network (Faster R-CNN) is adapted with transfer learning for detection of workers and pieces of construction equipment on the jobsite, while the Simple Online and Realtime Tracking (SORT) approach is applied for object tracking. A hybrid model integrating CNN and Long Short Term Memory (LSTM) is employed for action recognition. An alternative form of the Crew-balance Chart (CBC), called line chart in which anomalies are pre-screened, is utilized for recognized actions. Validation was carried out with earthmoving operations. The trained Faster R-CNN reached a 73% Average Precision (AP), and the SORT algorithm modified by this work successfully reduced identity switches. Irregular operations in the testing videos were identified, and truck exchanges were filtered. In addition, an activity log was produced with basic information along with starting and ending times of the identified irregular operations. With the line chart and the log provided by the proposed framework, field managers can efficiently identify potential abnormal activities, providing the opportunity for further investigations and adjustments accordingly.

Introduction

For the construction industry, maintaining high performance in attempt to reduce project delay is a common goal [1]. Meeting project deadlines is the most critical aspect for the owners during construction [2]. Various circumstances may compromise productivity or even cause construction suspension; consequently, identification of abnormal activities in a timely fashion is vital for effective project control. To discover the anomaly, construction process analysis is indispensable. Typical approaches for this goal include on-site supervision and monitoring of surveillance videos, which are labor-intensive and time consuming.

With the booming popularity of the Internet of Things (IoT) and artificial intelligence, numerous studies have provided applicable frameworks to automate construction progress monitoring [[3], [4], [5], [6], [7]]. For example, Inertial Measurement Units (IMUs) were installed on heavy equipment to measure cycle times [8], depth sensors were applied to capture motion data, and the Support Vector Machine (SVM) and Hidden Markov Model (HMM) were adopted for worker action classifications [9]. Tang et al. [10] not only applied Faster Region-proposal Convolutional Neural Network (Faster R-CNN) combining Feature Pyramid Network (FPN) to detect construction resources, but further took interaction between workers and construction resource instances into consideration for safety improvement. In addition, Crew Balance Charts (CBCs) were used to visualize construction activities for field managers to obtain comprehensive information [11].

While existing studies have established automatic monitoring systems [[3], [4], [5], [6], [7], [8], [9], [10]], most of these works focus on the methodologies of construction resources detection, action and activity recognition, but detection of abnormal events. In other words, there are still opportunities to improve activity monitoring and analysis. This paper proposes an image-based analytics framework to automatically identify irregular activities through an alternative form of the CBC, called line chart. In this line chart, operation cycles which are irregular compared with standard operations are identified and highlighted. Meanwhile, a log is created with starting and ending times of each irregular event. Irregular operations are potential abnormal events that may decrease productivity and may be related to construction safety. With the provided information, field manager can directly acquire surveillance videos of irregular operations to investigate the cause of anomalies.

The remaining sections of the paper are summarized as follows. Section 2 reviews the related works. Section 3 states the pipeline of image analytics. Section 4 presents the quantitative results. Section 5 discusses the outcomes of image analysis, contribution and limitations of this study. Finally, Section 6 concludes the work.

Section snippets

Literature review

Progress monitoring on the construction jobsite is vital to the overall project delivery. Recognition of construction activities provides the opportunity for more detailed analysis of the process. Numerous studies adopted IoT sensors, such as IMUs or accelerometers, to collect essential action data [6,8,12]. However, installation of those devices on every worker and piece of equipment could be costly and inconvenient [13]. In addition, workers may feel uncomfortable or inconvenient to wear

Methodology

This study follows the typical pipeline consisting the following four steps: object detection, object tracking, action recognition, and operational analysis (Fig. 1). The objects of interest, such as workers, excavators and dump trucks, are detected in images. Second, detection results are associated in attempt to distinguish individual trajectories. In other words, tracking is conducted. Third, the time-series data obtained from the previous module are used to recognize particular actions.

Experimental results

Although several studies regarding automatic construction monitoring were conducted, little data has been provided for public usage. The earthmoving operation dataset, with excavators and dump trucks, provided by Roberts and Golparvar-Fard [5] is available, and this section exclusively focuses on these activities.

Discussion

In this section, insights regarding the analysis are provided, and the contribution and limitations of the work are discussed.

Conclusion

This study proposes an image analytics approach for field managers to examine irregular activities which may detain the schedule. The framework consists of four modules: object detection, object tracking, action recognition, and operational analysis.

For object detection, the pre-trained Faster R-CNN was adopted to recognize objects in the image taken from construction sites. The model goes through transfer learning with the ACID for better detection results.

For object tracking, both SORT and

Declaration of Competing Interest

We declare no conflict of interest to any individual.

Acknowledgment

The authors would like to thank the Research Center for Building Information Modeling and Management at National Taiwan University for the support that has made this work possible. The second author would also like to thank the Ministry of Science and Technology (MOST) of Taiwan for the support of grant MOST 109-2221-E-002-015.

References (74)

  • H. Luo et al.

    Convolutional neural networks: computer vision-based workforce activity assessment in construction

    Autom. Constr.

    (2018)
  • L. Ding et al.

    A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory

    Autom. Constr.

    (2018)
  • J. Cai et al.

    Two-step long short-term memory method for identifying construction activities through positional and attentional cues

    Autom. Constr.

    (2019)
  • Q. Fang et al.

    Detecting non-hardhat-use by a deep learning method from far-field surveillance videos

  • W. Fang et al.

    Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach

    Advanced Engineering Informatics

    (2018)
  • M. Yang et al.

    Temporal dynamic appearance modeling for online multi-person tracking

    Comput. Vis. Image Underst.

    (2016)
  • O. Angah et al.

    Tracking multiple construction workers through deep learning and the gradient based method with re-matching based on multi-object tracking accuracy

    Autom. Constr.

    (2020)
  • A. Justel et al.

    A multivariate kolmogorov-smirnov test of goodness of fit

    Statistics & Probability Letters

    (1997)
  • M.C. Gouett et al.

    Activity analysis for direct-work rate improvement in construction

    J. Constr. Eng. Manag.

    (2011)
  • J.J. Lin et al.

    Visual data and predictive analytics for proactive project controls on construction sites

  • A. Khosrowpour et al.

    Automated worker activity analysis in indoor environments for direct-work rate improvement from long sequences of rgb-d images

  • K. Liu et al.

    Crowdsourcing construction activity analysis from jobsite video streams

    J. Constr. Eng. Manag.

    (2015)
  • E.R. Azar et al.

    Part based model and spatial–temporal reasoning to recognize hydraulic excavators in construction images and videos

    Autom. Constr.

    (2012)
  • T.-Y. Lin et al.

    Focal loss for dense object detection

  • K. Kang et al.

    Object detection from video tubelets with convolutional neural networks

  • D. Roberts et al.

    Vision-based construction activity analysis in long video sequences via hidden markov models: Experiments on earthmoving operations

  • K. Hara et al.

    Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and Imagenet?

  • K. He et al.

    Deep residual learning for image recognition

  • D. Roberts et al.

    Vision-based construction worker activity analysis informed by body posture

    J. Comput. Civ. Eng.

    (2020)
  • J. Redmon et al.

    Yolov3: An incremental improvement

  • J. Carreira et al.

    Quo vadis, action recognition? a new model and the kinetics dataset

  • A. Newell et al.

    Stacked hourglass networks for human pose estimation

  • J. Gong et al.

    Computer vision-based video interpretation model for automated productivity analysis of construction operations

    J. Comput. Civ. Eng.

    (2010)
  • A. Zimek et al.

    Outlier Detection

    (2017)
  • X. Yang et al.

    Outlier Detection with Globally Optimal Exemplar-Based GMM, in: Proceedings of the 2009 SIAM International Conference on Data Mining

    (2009)
  • M. Pavlidou et al.

    Kernel density outlier detector

  • R. Chalapathy et al.

    Deep learning for anomaly detection: A survey

  • Cited by (27)

    • Digital technology for quality management in construction: A review and future research directions

      2022, Developments in the Built Environment
      Citation Excerpt :

      CV has been used for construction activities identification (Seo et al., 2015), site productivity assessment (Luo et al., 2018), detection and visualization of dynamic workspaces (Luo et al., 2019), and real-time structural health (Dong and Catbas, 2021). Visual data collected from construction sites are used for site safety (Fang et al., 2020b), abnormal construction activity identification (Lin et al., 2021), and automated postconstruction quality assessment for defects in inspection (Liu et al., 2017). Blockchain is defined as a distributed ledger technology characterized by decentralized operations across a consensus mechanism network (i.e., peer to peer), where all data are stored as blocks that are immutable once joined and authenticated in a chain (Viriyasitavat et al., 2019).

    View all citing articles on Scopus
    View full text