Construction machine pose prediction considering historical motions and activity attributes using gated recurrent unit (GRU)

Highlights

• Proposed a GRU-based method for machine pose prediction using motion capture data

• Improved machine pose prediction results by keypoint-based activity recognition

• Considered non-geometric data like machine interaction and activity type

• A rollback method greatly reduced the influence of uncertainty in activity recognition

• Achieved an average accuracy over 90% for machine pose prediction

Abstract

The variation of construction machine poses is one of the main causes for interactive on-site safety issues such as struck-by hazards. With the aim to reduce such hazards, we propose a framework for predicting construction machine poses based on historical motion data and activity attributes. After building a machine motion dataset, we develop a keypoint-based method for recognizing machine activities considering working patterns and interaction characteristics. The recognized activity information is then incorporated with historical pose data to predict future machine poses through a type of recurrent neural network (RNN), named Gated Recurrent Unit (GRU). In experiments of using excavators as the objects, our framework achieves decent performance for machine pose prediction, which is further improved by incorporating activity information, reaching an average percentage of correct keypoints (PCK) of 90.22%. The results indicate the high potential of our framework in predicting construction machine poses and improving on-site safety.

Introduction

Construction sites are suffering from high hazard rates among all workplaces, making it a foremost concern to improve the on-site safety. According to the reports from the U.S., mainland China and Hong Kong Special Administrative Region [[1], [2], [3]], unsafe operation of construction machines is an essential reason of fatal hazards occurred on construction sites. In the U.S., over 38% of construction accidents are caused by interactions between construction resources (e.g. workers and machines), which also resulted in more than 16% and 29% of construction accidents in mainland China and Hong Kong SAR, respectively. Hence, it is important to monitor the motion of construction machines on sites. Typically, accidents are likely to take place when two on-site objects move towards each other, making it important for safety managers to pay attention to the changing locations (i.e. trajectories) of on-site construction resources. In recent years, several efforts have been made to automatically monitor locations of construction resources based on data captured from surveillance cameras and pre-installed devices to assist site managers with traditional error-prone and tedious safety inspections of construction sites [[4], [5], [6]]. Nevertheless, it is common but easily overlooked that accidents may still occur when the location of heavy construction machines remains unchanged, but their poses are varying constantly as deformable components of construction machine are operated. Therefore, monitoring machine poses can be a necessary supplement for safety management of construction projects.

Up till now, most site managers monitor on-site machine poses by watching surveillance videos and evaluating potential risks manually. Such manual observations of on-site safety condition are error-prone and time-consuming because they are greatly dependent on physical status and expertise of the inspector. To address such limitations, previous studies have attempted to automate pose monitoring of construction machines based on surveillance videos. For example, several efforts have been contributed to estimating past and current poses of construction machines through processing on-site videos not only using conventional computer vision techniques [7], but also further adopting deep learning techniques [8,9] that have shown promising performance in many vision-based tasks. Besides, another common and practicable approach for automating pose monitoring of construction machines is processing the collected signals from pre-installed devices [10]. Most of previous studies only focus on estimating past and current machine poses which have occurred, yet understanding the current poses of construction machines is not sufficient to avoid potential hazards. Instead, pose prediction (or pose forecasting) [11], of construction machines can provide more clues to prevent possible collisions or other accidents, for which there is still a lack of research.

When tentatively investigating potential methods for predicting future poses, we found that both geometric and non-geometric information provide insights for prediction. On the one hand, future poses of construction machines are influenced by machine motions which can be informed by geometric construction data, such as the geometry information of machines as well as the construction environment. On the other hand, non-geometric construction data, such as the working task the machine is focusing on and the interaction of the machine with other objects, can provide contextual information for predicting machine poses, which is not fully considered in other research. With the aim to reduce potential on-site hazards, we propose a framework incorporating both geometric and non-geometric construction information to improve the performance of machine pose prediction, where geometric construction data refer to the dimensions (e.g. length, width, height) and coordinates of both construction machines and project terrain, while non-geometric data are semantic data such as working tasks and working natures of the construction machine. The proposed framework consists of three modules, i.e. motion capture module, activity recognition module and machine pose prediction module. Firstly, geometric construction site information is included in the machine motion capture module, and thereon historical motion data of target construction machines are provided. Next, the machine activity recognition module recognizes historical activities with the help of non-geometric construction information and historical motion data from the machine motion capture module. Lastly, the machine pose prediction module generates future poses of construction machines by integrating information of historical poses and activities. To validate the proposed framework, we adopted excavators as experiment objects as excavators own more deformable components and tend to have complex pose variations and interactions with the surrounding objects, and performed experiments based on a motion capture dataset proposed by adding historical poses data to an existing video dataset. The proposed framework is expected to provide more comprehensive information to reduce potential hazards caused by varying machine poses and to protect on-site workers.

The rest of this paper is summarized below. Section 2 reviews works related to motion analysis of the past and current states of construction machines, as well as machine motion prediction. Afterwards, Section 3 introduces the proposed overall framework of machine pose prediction that considers both geometric and non-geometric construction information, including detailed descriptions and implementation approaches of each framework module. Next, taking excavators performing earthmoving tasks as the example, the proposed overall framework is verified in Section 4, and thereon discussion and insights about the experiment results are given. In the end, conclusion is stated in Section 5.