Hybridization of reinforcement learning and agent-based modeling to optimize construction planning and scheduling

Highlights

• Implementation of agent-based simulation in construction processes

• Optimization of construction project scheduling

• Hybridization between reinforcement learning, agent-based simulation modeling and graph embedding methods

• Practical solutions to aid decision-making processes in construction project activity sequencing and work-breakdown formation

Abstract

Decision-making in construction planning and scheduling is complex because of budget and resource constraints, uncertainty, and the dynamic nature of construction environments. A knowledge gap in the construction literature exists regarding decision-making frameworks with the ability to learn and propose an optimal set of solutions for construction scheduling problems, such as activity sequencing and work breakdown structure formulations under uncertainty. The objective of this paper is to propose a hybrid reinforcement learning–graph embedding network model that 1) simulates complex construction planning environments using agent-based modeling and 2) minimizes computational burdens in establishing activity sequences and work breakdown formations. Three case studies with practical construction scheduling problems were used to demonstrate applicability of the developed model. This paper contributes to the body of knowledge by proposing the hybridization of reinforcement learning and simulation approaches to optimize project durations with resource constraints and support construction practitioners in making project planning decision-making.

Introduction

Construction planning and scheduling is the process of determining what activities are performed and establishing how and when these activities are conducted within the limits of the available time, budget, and resources [1]. According to the Project Management Institute (PMI), planning activities consists of transforming the scope of work to establish a hierarchy of manageable work packages, also called a work breakdown structure (WBS) [2,3], and then determining the sequence of activities' execution according to project constraints including work environment layout, available resources, and scope. In the same manner, construction planning enables a project to accomplish a set of required objectives that can be considered as a two-part problem. First, the solution needs to capture the dynamic construction environment with activities representing project scopes that can be defined as a hierarchy of executable work packages. Second, the solution is a result of estimating duration requirements for activities and optimizing activity sequencing based on multiple and pre-determined constraints that also incorporate decision makers' knowledge and experience. Construction planning includes scheduling and other forms of planning, such as material handling, site layout planning, equipment path planning, and site logistics planning [4]. Scheduling problems are an important part of construction planning activities in terms of planning physical construction project components that have a specified set of start and finish timelines and an estimated duration.

Researchers have proposed multiple decision-aid methods, such as simulation, optimization, multi-criteria decision-making, and automation, to tackle activity sequencing and WBS formations in construction scheduling problems [4]. Some methods include linear programming, heuristic or meta-heuristic approaches, and hybrid simulation approaches such as discrete event simulation-genetic algorithm (DES-GA). These methods have proposed solutions by solving mathematical objective functions that optimize a given metric, such as time, cost, resource, or quality. These approaches have some shortcomings in capturing uncertainty in the construction environment, raising computational burdens, and not being easily generalizable to multiple construction projects. In a scheduling problem, the optimization process needs to consider multiple constraints tied to each activity, such as time, budget, and resources. These constraints can include 1) precedence relationships, 2) project manager preferences, such as activity associated with a rented crane may need to take precedence to minimize equipment rental costs, and 3) interruptions, such as equipment breakdowns. To tackle these constraints, methods are needed that can capitalize on the simulated environment to understand complex behaviors and derive more sufficient decisions.

Reinforcement learning (RL) is very effective for decision-making processes in construction problems. RL algorithms are able to solve optimization problems with higher constraints [5] and perform efficiently with increasing complexity and number of activities [6]. The RL agent learns to implement better actions, including optimal sequencing of activities, through training achieved from exploiting local rewards and exploring random actions despite lower rewards. Hence, RL can help fill the aforementioned shortcomings of current decision-aid methods in construction planning by developing a local decision-making policy for each agent, based on communication channels, and by breaking down the problem into sub-problems, all of which contributes to computational efficiency. Using RL assists construction practitioners in facilitating generalizations through the learning process, because different problems can be broken down into similar sub-problems. Moreover, RL facilitates agent communications and enables agents to arrive at a set of decisions involving a set of joint actions. This results in a faster convergence to the optimum global policy. However, an RL process does not capture the dynamic nature of modeling in the construction environment, because of the complexity caused by various interactions between system components [7]. In a construction setting, however, having a model of the construction environment is crucial.

Simulation techniques have been used to capture the dynamic nature of the construction environment as well as uncertainties in the modeling process [8]. Compared to other simulation techniques, such as DES and system dynamics (SD), agent-based modeling (ABM) is able to handle these complexities and capture emerging behaviors. ABM is capable of handling very complex real-world systems often containing large amounts of autonomous, goal-driven, and adapting agents [9]. ABM uses a bottom-up approach where the system is described as interacting objects with their behaviors, which allow complex emergent behaviors to be captured. ABM enables tracking of agent interactions in their artificial environments to understand overall processes that lead to global patterns [10]. By incorporating ABM in an RL process, necessary features that support environment modeling, such as system parameters, system behaviors, and rules, are provided in order to enable an efficient representation of the dynamic construction environment and provide the RL platform with the necessary features to support environment modeling.

The objective of this paper is to propose an RL-ABM method with graph networks that can be used to support decision-making in construction planning by providing optimum work package sequencing to schedule activities based on project constraints. The application of the proposed model can be extended to establishing a WBS for a construction project. Three case studies were used to demonstrate the proposed model and discuss the applicability of RL-ABM to addressing similar problems related to activity sequencing. The developed RL-ABM method enables construction decision-makers to evaluate project objectives, facilitates the optimization of multiple types of resources during planning through the RL agent's learning ability, is able to incorporate resource planning during schedule development, and can be generalized to other construction planning problems. Moreover, the applications of the method can be extended to scope definition (WBS formulation) at the project level in future work that will extend this study.

The rest of this paper is structured as follows. First, as background, a literature review section is presented, which discusses decision-making in construction planning and shortcomings of current decision-aid approaches to scheduling problems, followed by an introduction of simulation approaches and RL to address the gap in the literature. Next, the theoretical development of RL-ABM is presented as part of the proposed methodology, which also includes the steps of problem definition, ABM simulation, and development of the RL model. Three case studies are then presented to demonstrate application of the proposed RL-ABM method. Finally, conclusions are presented and recommendations for future work are discussed.