A vision-based approach for automatic progress tracking of floor paneling in offsite construction facilities

Highlights

• A novel framework to automatically detect and track progress of offsite construction operations.

• A deep learning and finite state machines hybrid method is applied to CCTV footage of floor paneling workstations.

• Task durations, utilization of resources, and production efficiency per panel are estimated with high accuracy.

Abstract

Offsite construction is an approach focused on moving construction tasks from traditional jobsites to manufacturing facilities. Improved productivity of construction tasks is paramount in terms of competitiveness and is achieved through the continuous improvement of operations and planning, which often relies on historical data obtained from previous projects. Despite being a common practice, current methods, such as time studies, are not able to capture the changing scenarios resulting from improvements to production. This paper presents a novel approach to automatically detect and track the progress of construction operations by applying a method that combines deep learning algorithms and finite state machines to existing footage captured by closed-circuit television (CCTV) security cameras. Applied in the context of floor panel manufacturing stations, the proposed method examines entire production days recorded by CCTV cameras, while providing the durations of each task, its required resources, and the task efficiency per panel with high accuracy.

Introduction

Offsite production is an increasingly popular approach in construction that relocates most on-site operations to a more controlled factory environment. Offsite construction facilitates a hybrid approach that can be described as a series of construction operations in an assembly line, which offers several advantages over traditional construction such as mass customization, increased productivity and quality, improved safety and health for construction workers, minimization of construction waste, and shorter delivery times [1]. The competitive advantage comes from the adoption of automation, innovative facility layouts, and broad adoption of information technology. The performance of offsite facilities depends on labor and production planning because poor planning can translate into bottlenecks on the production line and an increase in production costs [2]. Consequently, delays create a gap between planned production and actual output, which prevents offsite practitioners from meeting their scheduled commitments. These delays are associated with inaccurate productivity metrics for floor panel production due to the high degree of variability in the cycle time at the workstations resulting from variations in the panel design specifications and due to the lack of accurate data collected from the production line [3]. Despite significant performance improvements at offsite construction facilities, studies aimed at productivity improvement are based on employing manual observations to monitor construction activities, a method that is error-prone and often provides wide ranging of results [4]. Furthermore, manual work monitoring limits productivity enhancements in offsite construction, since it only captures the state of production at the time it is performed and is not sustainable for long-term planning of operations due the ever-changing nature of improvements performed at the facility.

Systematic monitoring of construction operations can bring an immediate awareness of task-specific issues [5,6]. It provides stakeholders with necessary information related to scheduling, costs, productivity, and resource utilization, that quickly supports project control decision-making [7]. Presently, a large number of data collecting technologies are used for progress tracking, from radio-frequency identification (RFID) to aerial photogrammetry [8]. The selection of the correct technology, however, depends heavily on the project requirements and the level of readiness of the required data [9]. As a type of easily captured and widely spread media, images and videos have become popular in the architecture, engineering and construction (AEC) industry. Applying vision-based technologies to analyze the recorded images and videos automatically has drawn much attention from practitioners and from academia [10]. Several interdisciplinary works have enabled the measuring, detecting and tracking of objects, i.e. equipment and/or workers, which play a critical role in construction performance monitoring applications [11]. However, most studies focus on on-site activities [12], which means the proposed solutions are hardly applicable to the offsite construction sector [9]. Indeed, Arashpour et al. [1] indicate the deficiency of on-site production tracking systems for use in offsite construction facilities due to the fundamental ways the offsite approach differs from the on-site approach, such as the accelerated pace at which the trades work in manufacturing workstations and the long-term production targets (e.g., daily, weekly, or even monthly quotas), thus, requiring short- and long-term production tracking at the same time. This information is required to monitor production deviations throughout the day and provide real-time information so management can address bottlenecks in a timely manner while being aware that these deviations will have an impact to some degree on production in the following days. This requires tracking systems that provide consistent, reliable, and accurate data for short and long periods of time.

Whereas site-built methodologies focus on identifying the key resources among the crowded job site to determine the current activity or scheduled task, i.e. workers/crews [[13], [14], [15]] or heavy equipment [16], offsite facilities consist of workstations, each with limited and sequenced activities that should be easier to identify. However, compared to construction sites where cameras are temporarily located and oriented specifically for the monitoring task at hand, cameras are already installed in offsite facilities for closed-circuit television (CCTV) security footage of the facility. Such cameras typically provide low-resolution videos to facilitate long-term storage. In fact, low-resolution cameras are recommended for workplace surveillance due to privacy issues and workers' mental health [17,18]. However, the reported performance levels of state-of-the-art activity monitoring methods using such cameras are below 50% in terms of accuracy [19]. To overcome the low resolution environment and enable automatic monitoring of tasks in offsite construction facilities, this paper proposes a hybrid method that combines deep learning algorithms and virtual finite state machines (VFSM). This approach uses the accurate vision-based detection and classification of novel deep learning algorithms, such as the Faster Region-based Convolutional Neural Network (R-CNN), along with robust computationally modeled transitions between sequences of events, for which VFSM is a widely used technique in the robotics field [20]. Moreover, this approach provides a novel method to capture productivity-related metrics, such as duration and number of workers per task, in order to address and enhance the efficiency of offsite construction operations. The methodology is tested and finally validated using video footage from a floor panel manufacturing workstation in an offsite construction facility.

Offsite construction is often associated with shortened schedules and increased productivity in different parts of the world [21,22], allowing on-site construction work, i.e. foundations, to be performed concurrently with the fabrication of the project's structure in the form of panels or volumetric modules. The increased productivity results from both external and internal factors: (1) a controlled environment immune to the influence of weather conditions [23], and (2) higher productivity in construction operations due to process improvement studies conducted at the factory. Indeed, offsite construction provides a more suitable environment for data collection, which allows practitioners to collect more reliable data due to the controlled environment and reduced variability of motions from workers [24]. Through interviews and observations, it is possible to reduce production hours considerably by creating a culture of continuous improvement and identification of value and non-value added tasks in the overall process [25], while applying other methods to predict other aspects of production. Sanderg and Bildsten [26] identify overproduction, waiting, and needless movements as the most critical waste activities in offsite construction production because they render workstations idle and non-productive.

Different techniques are applied to address value in offsite construction tasks such as value stream mapping, which identifies opportunities for improvement to reduce production lead times [27], and fuzzy analytic hierarchy processes, which rank expert opinions to estimate the risks associated with delays and cost overruns from the project's initial baseline [28]. From these analyzes, several methods are employed to improve construction tasks and improve productivity at the manufacturing facility. Mullens and Kelley III reduces the task's cycle time by eliminating non-value-added motions at workstations and by constantly monitoring production output [29]. A dependency structure matrix is applied to prevent bottlenecks, rework, and congestion at stations to reduce the number of main activities by 30% through the observation of construction tasks and their inter-dependence [30]. The impact of multi-skilled labor in offsite construction facilities is also addressed, taking into consideration various strategies for its implementation and quantitative indicators pertaining to productivity and cost [31]. Other studies consider a holistic approach to quantify the economic, social, and environmental aspects of production to suggest improvements and maintain a continuous improvement culture in the company [25].

Despite the relevant improvements recommended in the aforementioned publications, all input data is collected in the form of time studies of construction tasks, usually taken manually, which is an error-prone, monotonous, tedious, and time-consuming process. Moreover, manual time studies fail to address the impact of changes on the production line, thus becoming obsolete after a meaningful improvement of a task [32], requiring a new effort to collect updated data for further improvement. This poses as a significant barrier to continuous improvement in offsite construction facilities over the long term without significant investment and commitment from the companies. In order to address this issue, the use of real-time tracking systems, such as RFID and barcodes, are proposed to provide real-time feedback to upper management regarding inventory and production while maintaining a permanent method of communication between production and process improvement teams [33].

Presently, applications using real-time tracking systems are scarcely employed in offsite construction, especially in the context of managing production lines. Nevertheless, cost savings and increased productivity are already observed in offsite construction facilities when using barcodes to track material flow and inventory [34]. Moreover, production schedules are optimized using RFID tags when taking into consideration wall panels' attributes, job sequencing, and constructability aspects from production [35]. RFID systems are also applied in quality and production tracking, providing a system for proactive process improvement on production lines [36]. Despite the benefits involved in this approach, doubts regarding the implementation of real-time tracking systems are raised due to the inherent trade-off between the cost to implement these technologies and any quantifiable gains in productivity. Anderl and Fleischer [37] argue the initial investment in hardware limits a broader application of these systems, thus necessitating alternative approaches to implementation. In order to provide an alternative solution for the implementation of real-time tracking systems in offsite construction facilities, this paper proposes the use of computer vision techniques to track process and idle times from existing CCTV security footage captured by cameras installed to monitor a production line.

Despite growing interest and relevant work in the past twenty years, there is a bias in research literature towards traditional construction over offsite construction involving the application of computer vision [10][]. Studies involving the application of computer vision in offsite construction are scarce across several research areas. Martinez et al. propose a system to perform real-time quality inspections of light steel-framed panels by comparing the image of the assembled panel with its intended design [38,39]. Moreover, algorithms are developed to detect the work progress of wall panels during construction, taking into consideration different components of walls at different stages [40]. Planning and final installation of manufactured elements at their designated sites are also the subject of several studies, such as the combination of various information systems and computer vision for the installation of structural elements [41], and the identification of images to detect module lifting tasks for the final assembly of high-rise buildings [42].

The application of computer vision in traditional construction environments has advanced work monitoring through the automatic detection of labor and work progress in a rapid and accurate manner [43]. Construction tasks and finished products are similar in the context of both traditional and offsite construction, which provides interesting insights for work monitoring in the latter scenario. However, the significant differences in the physical spaces must be taken into consideration for accurate analysis and are addressed in this paper. A significant challenge for image processing in traditional construction scenarios is the constant change of reference from viewpoints and the Region of Interest (RoI), the area of the image from which information is extracted, due to objects obstructing the camera on-site [44]. Moreover, constant changes in illumination pose a significant challenge in terms of the accuracy of computer vision algorithms [45], while the installation of permanent tracking devices on traditional construction sites poses challenges, such as the need for electricity and concerns about theft [46]. Overall, from a computer vision perspective, these challenges are significantly reduced in an offsite construction facility: 1) workstations are fixed and worker motions are more predictable according to the factory layout; 2) lighting conditions are stabler throughout the day, and sun glare and shadowing effects are minimum; and, 3) cameras are permanently installed at fixed spots, free of obstructions due to their original goal (security), meaning they would not require relocation nor would there be any concerns about theft.

Several studies apply support vector machines (SVM) and other computer vision algorithms, such as EDLines and line segment detectors (LSD), to identify building elements (walls, equipment, etc.) and their progression during the project based on the images collected on-site in order to identify the progress of activities during construction [[47], [48], [49]]. Despite being a good solution for identifying the progress of a project, non-permanent data collection of images inhibits further productivity analysis at the activity level since the number of workers and time spent on each activity is not recorded. Hence, to monitor the work and productivity of construction tasks at an offsite construction facility, computer vision algorithms must be able to track workers and recognize the task being performed through video to monitor both progress and productivity despite the significant extra effort to process videos instead of static images. Gong and Caldas [49] describe the main three approaches for automated productivity measurement using video images: (1) trajectory recognition and tracking of resources, (2) movement detection of construction resources, and (3) recognition of worker's gestures. Trajectory recognition may not be very applicable in the context of automated productivity measurement in offsite construction since the tasks are contained within each workstation, and internal trajectories of workers are not without purpose. Therefore, movement detection of equipment (e.g., cranes, multi-function bridges, etc.) and recognition of worker's gestures are more suitable approaches for the implementation of computer vision algorithms to measure productivity in offsite construction automatically.

Despite the positive results, Luo et al. [52] argue that the changes in background, which can include image obstruction, illumination variations, and viewpoint changes, combined with multiple interactions and group activities reduce the accuracy of the proposed methods, thus suggesting the application of convolutional neural networks (CNN) to overcome these issues and automatically recognize workers' activities. CNNs can accommodate complex tasks while simultaneously capturing static, short-term, and long-term motions in a video from large datasets in a timely manner [50]. In fact, CNN architectures have been identified as the main support for robust monitoring of construction resources [51,52], and have been applied in a number of different areas, such as safety [53], progress monitoring [54], and resource localization [55]. This is because neural networks, including CNNs, have superior capabilities in terms of representing complex relationships between inputs, i.e., images, and desired outputs, i.e., resources (objects). Most recent frameworks with the aim of monitoring resources on job sites include CNNs for the initial object detection [56,57].

Continuous monitoring of construction projects allows project managers to evaluate the operational efficiency of their processes and input resources (i.e. crew production rates), determine risk factors that can cause delays or safety accidents, and analyze current construction progress [58]. R-CNNs have been recently used to detect various types of construction objects, including workers and equipment to enable such close monitoring of operations. Fang et al. used such approach to provide efficient monitoring of heavy equipment and operators [19]. Kim et al. trained a similar neural network to monitor the construction productivity in an earthmoving process [59]. Luo et al. built an activity recognition method, based on CNNs, to detect multiple construction resources and interpret their spatial relationships in order to estimate the operational efficiency of the construction processes and resources used [60]. Similarly, job site crowdness and the possibility of potential dangerous physical interferences between resources have been proposed to be monitored by using R-CNNs [55,61]. Overall, deep learning algorithms have shown great performance on vision-based construction monitoring. For this study, the authors follow the current trend of incorporating CNN architectures for the purpose of identifying resources within offsite construction facilities and providing robust data to monitor repetitive tasks.

A finite state machine (FSM) is commonly used as a control strategy for physical devices and, as a mathematical construct, is used to approximate a broad range of physical or abstract phenomena [62]. A FSM aims to control a sequence of actions depending on certain triggering events or rule sets, where the number of actions or states is a predefined finite list. The FSM changes its state when a condition is satisfied through a state transition. A FSM can be mathematically defined as a quintuple (K, S, S₀, C, S_F): (K) is the set of events, which acts as an input of the FSM; (S) is the predefined, finite, and non-empty set of states; (C) is the state-transition functions, C = K × S; and (S₀) and (S_F) are the initial and final states, respectively, which are subsets of (S). A virtual finite state machine (VFSM) is a finite state machine that is completely defined and executable in a virtual environment. A VFSM offers an alternative to rule-based algorithmsto describe the behavior of a system or process without any physical interference or action [63]. VFSMs have been considered a more robust and transparent approach to represent dynamic systems in comparison to rule-based systems, especially considering the strength of having stable states that minimize the potential of erroneous identification. Targeting the automation of any process, the FSM has played a key role in the modeling and testing of autonomous equipment for the past several decades [64,65].

In construction, however, the integration of state machines in autonomous or automatic consruction processes is rare. A few recent examples can be found in the autonomous control of critical equipment, i.e., slurry shield tunnel boring machinery [66], or modeling of construction equipment activities, i.e., trucks and excavators during earthmoving operations [67]. In general, modeling through state machines has been overlooked by the academic community in regard to construction activities. Researchers in the construction field prefer to model construction operations using discrete-event simulation (DES) models, which are especially popular to monitor construction operations on job sites [68]. While both methods rely on consistent and accurate input data to build a correct model, DES and VFSM yield different benefits in the long term. DES provides after-de-facto modeling and simulation of different scenarios to assist in determining which future actions will improve productivity. Meanwhile, VFSMs can be executed and provide monitoring data in real-time (e.g. task durations) while storing this information to address the task efficiency after it is finished or compare its data with similar cases. From a construction management perspective, DES modeling synchronizes adequately with the project-centered, schedule and cost-driven construction mentality [69], and justifies the academic focus on this modeling approach. VFSM, on the other hand, is able to instantly provide data that can be analyzed to determine the productivity and reliability of operations (e.g. task efficiency) in real-time based in the system's change of states which are not, in most cases, of high interest to construction managers [70]. Moreoever, offsite facilities provide an interesting hybrid environment between manufacturing and construction that can benefit from both approaches. A few frameworks that focus on employing DES in offsite construction already exist in the literature: for example, to plan offsite construction projects [71], or to model inventory in panelized construction facilities [72]. This paper demonstrates the potential of VFSM as a modeling tool for construction activities, namely in the offsite construction industry.