Mobile projective augmented reality for collaborative robots in construction

Highlights

• Proposed a mobile projective AR framework achieving glassless AR that is visible to the naked eye.

• Designed algorithms for consistent projection on a job site using a mobile camera-projector.

• Prototyped mobile projective AR of BIM information on planar and non-planar surfaces.

• Demonstrated centimeter level projection error in laboratory settings.

• Enabled new human-robot information collaboration possibilities on construction sites.

Abstract

Augmenting virtual construction information directly in a physical environment is promising to increase onsite productivity and safety. However, it has been found that using off-the-self augmented reality (AR) devices, such as goggles or helmets, could potentially cause more health, safety, and efficiency concerns in complex real-world construction projects, due to the restricted field of view and the non-negligible weight of those devices. To address these issues, we propose a mobile projective AR (MPAR) framework in which the AR device is detached from human workers and carried by one or more mobile collaborative robots (co-robots). MPAR achieves glassless AR that is visible to the naked eye using a camera-projector system to superimpose virtual 3D information onto planar or non-planar physical surfaces. Since co-robots often need to move during the operation, we design algorithms to ensure consistent mobile projection with two major components: projector pose estimation and projection image generation. For planar surfaces, MPAR is achieved by a homography-based pose estimation and image warping. For non-planar surfaces, MPAR uses iterative closest point (ICP) for pose estimation and common graphics pipelines to generate projection images. We conducted both qualitative and quantitative experiments to validate the feasibility of MPAR in a laboratory setting, by projecting 1) as-planned building information onto a planar surface, and 2) as-built 3D information onto a piece-wise planar surface. Our evaluation demonstrated centimeter-level projection accuracy of MPAR from different distances and angles to the two types of surfaces.

Introduction

The construction industry has been labeled as hazardous [1,2] and susceptible to casualties and economic losses [3,4]. McKinsey [5] reported that the average annual productivity growth increased only 1% over the past 20 years, lower than 2.8% for the total world economy. Occupational Safety and Health Administration(OSHA) reports that the fatal injury rate in the construction industry is higher than the national average in any other industries in the US [6]. Some researchers believe these issues might be caused by lack of job site information for on-site construction operators [[7], [8], [9]]. To address the low efficiency and safety issues, AR has been applied to the construction industry [10] in different phases.

An AR system can help the collaboration and discussion between different parties, as it provides an interface that allows people to retrieve information when they are making decisions. One potential application is for designers to better communicate with customers about the interior design [[11], [12], [13], [14], [15]]. By displaying the virtual facility or decoration information generated by AR on a real site, clients can see exactly what a new piece of furniture or design will look like in their home before they purchase it. Lin et al. [16] designed a visualization system using AR technology to display the public building or facility information, facilitating the discussion process for different parties. To improve the interaction between field operators and the constructional information, Kim et al. [17] developed a system that can assist users in determining the optimum scenario for equipment operation by involving them in a shared AR environment.

Also, AR techniques have already been applied to visualize HVAC (heating, ventilation, and air conditioning) information [[18], [19], [20]]. Conventionally, when making HVAC renovation, construction workers take design papers to site and make a careful measurement to ensure they are making renovation in the correct place. With AR techniques, they can see directly what is behind the wall while working, e.g., drilling a hole according to the superimposed virtual information. AR can also assist the construction industry in facilitating employee training [21,22]. With the help of AR training platforms, construction workers can be trained with virtual materials, tools, or instructions, without being exposed to some dangerous training scenarios. Therefore, in conjunction with some other applications, such as hazard recognition and avoidance, AR has the potential to improve the safety for the construction industry [10,[23], [24], [25]].

However, current AR technologies have several disadvantages that could be problematic for architectural, engineering, construction, and facility management (AEC/FM). For head-mounted display (HMD) AR like HoloLens, the restricted field of view (FOV) might impede construction worker from obtaining environmental information [26,27] and the caused situational unawareness might result in fatal injuries to field workers [28]. Also, wearing a heavy helmet or goggles for long hours might cause occupational diseases for construction workers. Tablet-based AR can assist engineering processes by collaboratively visualizing computer-generated models [29]. However, one drawback is that construction workers cannot work while holding the tablet, and the inconvenience might limit the application of AR in construction. Also, Yan et al. [30] pointed out that tablet-based AR provides less immersive user experience compared to other AR technologies.

Therefore, one potential solution is to design a mobile augmented reality (MAR) system for AEC/FM, and Building Information Modeling (BIM) data retrieval, and construction education [[31], [32], [33], [34]]. Besides avoiding these disadvantages caused by HMD AR or tablet-based AR, a portable and mobile AR system itself will not be a burden to construction workers' workload. Also, the user can take the advantage of the MAR technique to assist construction collaborative robots in human robot interaction [35,36], by observing the context-related virtual objects augmented by the MAR system.

In this paper, we focus on the design and systematic evaluation of an MPAR system that uses a mobile projector system to blend virtual contents into real scenes while avoiding the problems mentioned above. This work is a continuation of our preliminary research [37]. Our method realizes AR with a camera-projector system and ensures a consistent projection during the working process, even if the projector is moved for various reasons (e.g., to enlarge/move the field of view, to make room for co-workers). The MPAR system benefits users in several aspects: 1. Users can place the MPAR system at arbitrary poses as long as the projection still covers the desired area. This is important because unlike in the classroom or movie theatre where projectors are strategically placed and fixed during the whole projection period, it is often difficult to ensure that the projector could stay static on a construction site with complex terrain or environment. 2. MPAR system is cost-efficient since it is only composed of a camera, a portable projector, and a computational platform such as a laptop. 3. It is easy to mount the MPAR system on a robot, such as an iRobot, such as the example shown in Fig. 1 and our video demonstration at https://youtu.be/rZE2RcduJ3E. In this prototype, the camera-projector system serves as the perception and projection module. The pan-tilt is used to control the pose of the camera-projector system. The Nvidia Jetson TX2 is the computational platform that can provide the functionality of computation, rendering, and human-robot interaction. The iRobot is the mobile base for the MPAR system. Such an MPAR system enables human-robot collaboration on construction job sites.

This paper first presents comprehensive applications or research of AR in the construction field, and related methods might be useful for our MPAR work. It then introduces the MPAR system's method in the methodology section. Finally, two experiments using the MPAR system are presented, and the quantitative evaluation work is also implemented to give an analysis for using the MPAR system.

Our contributions are listed as follows:

• We propose MPAR as a method to generate a proper projection image that can ensure that the projection result stays on designed locations, which is of fundamental importance for informational/cognitive collaboration between a collaborative construction robot and its human colleagues.

• We develop a mobile camera-projector system with both hardware and software that can achieve consistent projection results on both planar and non-planar surfaces during movement.

• We design a systematic way to automatically measure the projection accuracy on both planar and non-planar surfaces, which enables efficient comprehensive evaluation when deploying MPAR in real-world applications.