Approximate dynamic programming for network recovery problems with stochastic demand
Introduction
Infrastructure networks, such as transportation, power, telecommunication, and water, serve as lifelines for everyday activities of society. When disrupted, the functionality of these networks are degraded, adversely affecting the daily lives and economic productivity of communities (Miller et al., 2003, Foundations, 1997). For instance, a damaged transportation network will result in the obstruction of services that are delivered utilizing this infrastructure, such as the movement of goods and people (Berktas et al., 2016). Recently, after Hurricane Maria, Puerto Rico’s transportation network was severely damaged and as a result an estimated 10,000 crates filled with relief-aid supplies were stranded at the ports and could not be distributed for weeks (Domonoske, 2017). Hence, immediately after a disruption its imperative to recover the disrupted components of infrastructure networks as soon as possible to restore critical services.
In this work, we focus on road infrastructure networks and disruptions due to natural hazards with the objective of enabling the delivery of critical services as quickly as possible. However, the problem can be generalized to different types of disruptions and infrastructure networks. After a disaster hits, many infrastructure networks lose its functionality with the impact of the disaster (FEMA, 2007). Among these, road networks are disrupted mostly due to the debris accumulated on the roads. As a result, many critical services are hampered such as search-and-rescue operations and the movement of relief personnel and assets between supply and demand locations delaying the emergency response efforts. In order to enable the relief transportation between necessary locations, in the immediate response phase after a disaster, Federal Emergency Management Agency (FEMA) suggests that first roads should be cleared by pushing the debris blocking them to the curbsides (FEMA, 2007). In the rest of the discussion, we use the terms road clearance and recovery interchangeably.
For any problem concerned with infrastructure network recovery, including road clearance, developing an effective recovery schedule is a complex problem, mainly due to limited budgets, time and resource restrictions, and the network interactions. FEMA, in its most recent post-disaster guidelines, provides a high level suggestion on how to prioritize road clearance activities to develop a recovery schedule and suggests that roads should be prioritized based on their classes such that higher classes (such as major arterial routes, highways) take precedence (FEMA, 2007). However, this approach neglects the network interactions and the flow dynamics for providing services between critical network nodes, hence does not result in an effective policy when it comes to re-establishing operation of critical services in a timely manner (Ulusan and Ergun, 2018).
As in the case for many realistic applications of modeling and analytics, any decision support problem concerned with post-disruption operations, including road clearance, involves uncertain components. Uncertainty mainly arises due to the impeded information flow as a result of chaos and unpredictable human behavior in the aftermath of a disruption (Liberatore et al., 2013). The uncertain components in such problem settings include, but are not limited to, the amount of available resources for recovery, impact of the disruption, supply locations and demand amounts and distribution throughout the effected area (Bozorgi-Amiri et al., 2013). For instance, during the first several days after the Haiti earthquake the estimates for the number of disaster victims ranged from 30,000 to 100,000, indicating the extreme difficulty in providing precise need assessments (Van Wassenhove, 2006). Among all the uncertainties that may arise after a disaster, understanding and planning for demand uncertainty is critical since not correctly accounting for it may lead to sub-optimal or infeasible emergency response planning which may eventually lead to loss-of-life (Ben-Tal et al., 2011, Zhong et al., 2020). Motivated by the importance of addressing demand uncertainty and the documented difficulty in developing good need assessments in the immediate aftermath of a disaster, in this work, we consider the problem of developing effective road recovery policies given stochastic demand amounts for relief-aid commodities.
We define the stochastic road network recovery problem (SRNRP) as the problem of dynamically constructing a sequence of roads to clear to re-establish the time-sensitive service flow between critical nodes within the infrastructure network by establishing connectivity between relief-aid suppliers and demand locations with uncertain need levels. To the best of our knowledge, our work is first in the literature that provides a systemic approach to the road network recovery problem considering uncertain relief demand amounts.
We model SRNRP on a network where each edge (road) is associated with a given disruption intensity resulting in a debris amount on the road. Any road with non-zero debris is defined as a blocked road and needs to be fully cleared before any relief flow can travel through it. The number of clearance resources at each time-period is limited, hence establishing the connectivity between critical nodes in the network takes multiple time-periods. We assume that these clearance resources are located at the supply nodes and they can only clear roads that are reachable by an unblocked path from their location. For simplicity, we assume that a unit of debris requires a single clearance resource to be fully cleared.
The critical nodes in the SRNRP network are the relief-aid supply locations (e.g. hospitals, warehouses, points-of-distributions) and demand locations where there is a population in need of relief-aid. We assume that the location of all these critical nodes can be estimated with good precision and the supply nodes have known capacities, referred as their supply amounts. On the other hand, the demand amounts are stochastic with a given discrete probability distribution. The goal of the service operation is to enable the flow of services between supply and demand nodes in the shortest possible time through clearing the debris from the road network. Hence, we define the objective of SRNRP as maximizing the time dependent cumulative benefit of fulfilling demand, where a unit demand is assumed to be satisfied if it is connected to a service node via an unblocked path and a supply unit is from this node is allocated to it.
Markov Decision Process (MDP) is a powerful tool in modeling problems that are concerned with finding the best sequence of actions given the uncertainty in the environment. This algorithmic framework enables the determination of the optimal multi-period restoration policy in a post-disaster setting given the uncertainty in demand amounts. Hence, we model SRNRP as a MDP. We project the supply and demand information on the network nodes and the debris amounts on the network edges into the state variables in MDP, and define the set of feasible actions at each step of the MDP as the set of roads eligible for clearance. The reward gained after each action is completed corresponds to the benefit accrued by satisfying the demand as a result of the newly cleared road. The remaining reward to be gained from a particular state until the process terminates is referred as that state’s value or cost-to-go. The transition probabilities for reaching the next state from the current state are determined by the demand distribution. In general, MDPs seek to identify the best policy by prescribing an action to each state based on the state’s cost-to-go value with respect to different actions. In SRNRP terminology, given the supply, demand, and debris information over the disrupted road infrastructure network the MDP model seeks to find the best road to clear at each decision epoch to maximize the cumulative benefit accrued.
A significant concern when solving MDPs to optimality is the curse of dimensionality introduced by the large state spaces (Powell, 2007). For SRNRP MDP model, the intractable state space makes exact computation and storage of the state values impractical. To overcome this difficulty, we adopt an approximate dynamic programming (ADP) approach that relies on approximating the state value functions. In these approximations, we represent value functions as a linear combination of the states’ most salient properties, akin to a statistical linear regression. Through such an approximation, we aim to capture the most important characteristics of the states that determine their value functions and use these characteristics to derive close-to-optimal recovery policies.
The remainder of the paper is organized as follows, Section 2 provides an overview of the relevant literature and highlights the contributions of this work. Section 3 defines SRNRP formally and explains how we model it as an MDP. Section 4 further motivates the need to use an approximate method to solve the MDP and explains how we utilize domain knowledge to develop approximation models along with an algorithm to solve SRNRP. Section 5 describes the experimental setup including instance generation and various policies to be compared through the computations. Section 6 summarizes the results of various computational experiments and managerial insights, and Section 7 concludes the paper by highlighting the main insights, policy recommendations, and potential future research directions.
Section snippets
Literature review
We group the related literature into two streams based on (i) the problem setting, including papers that are broadly classified as network recovery problems in stochastic environments; and (ii) the proposed solution methodology, including papers that explore sequential decision making in stochastic environments through approximate dynamic programming.
Network recovery problem (NRP) considers the planning of restoration activities on interdicted infrastructure network components so that
Markov decision process framework for the stochastic road network recovery problem
In this section, first, we give a formal description of SRNRP and then explain how it is formulated as an infinite horizon MDP model.
We define SRNRP on a network , where node set consists of three subsets: (i) is the set of supply nodes with limited supply amounts, such as hospitals, warehouses, point of distributions or any facility providing the critical services, (ii) is the set of pre-determined demand nodes, such as locations in the disaster site with people in need of
Methodology
Various algorithms have been proposed for solving MDPs by determining optimal policies (Bertsekas et al., 1996, Puterman, 2014). Most of these algorithms entail solving the Bellman optimality equation in (6) to obtain an exact solution to an infinite horizon MDP problem. However, solving the Bellman equation presents several difficulties. The biggest challenge stems from computing Eq. (6) for all states. Notice that appears in both the left-hand side and the right-hand side of the equation.
Experimental setup
In this section, we present several ADP models and a benchmark algorithm that we computationally evaluated and compared across a set of instances.
In the MDP terminology the term policy refers to a mapping of each state to an action. However, any algorithm inducing a schedule of roads to clear over time may be interpreted as a policy. In the rest of the paper, we refer to the solution of any algorithm whether it is obtained by the ADP model, or generated heuristically as a policy, or
Computational results
In this section, we present results from a set of computational experiments designed to analyze and test the quality of the proposed ADP models. We summarize the main goals of the computational experiments as: (i) to evaluate the quality of the policies induced by the ADP on small-sized instances by comparing with optimal policies and to analyze the effect of selecting various basis function sets to define the ADP model; (ii) to analyze the impact of various problem characteristics, such as the
Conclusions
In this paper, we consider the problem of infrastructure network recovery when a disruption interdicts the network links used to deliver the necessary services. We model the problem on a road infrastructure network in a post-disaster environment where emergency response operations are hampered due to the debris blocked paths between critical emergency service providers, such as hospitals, and locations within the disaster site demanding services. In contrast to the works in the literature, we
Acknowledgment
This work is partially supported by the National Science Foundation (NSF) CMMI grant # 1537824.
References (64)
- et al.
Multi-vehicle synchronized arc routing problem to restore post-disaster network connectivity
European J. Oper. Res.
(2017) - et al.
Robust optimization for emergency logistics planning: Risk mitigation in humanitarian relief supply chains
Transp. Res. Part B
(2011) - et al.
Solution methodologies for debris removal in disaster response
EURO J. Comput. Optim.
(2016) Network restoration and recovery in humanitarian operations: Framework, literature review, and research directions
Surv. Oper. Res. Manag. Sci.
(2016)- et al.
Optimum post-disruption restoration under uncertainty for enhancing critical infrastructure resilience
Reliab. Eng. Syst. Saf.
(2019) - et al.
OR models with stochastic components in disaster operations management: A literature survey
Comput. Ind. Eng.
(2015) - et al.
A multi-stage stochastic programming model for relief distribution considering the state of road network
Transp. Res. B
(2019) - et al.
Incorporating network considerations into pavement management systems: A case for approximate dynamic programming
Transp. Res. C
(2013) - et al.
An approximate dynamic programming approach for the vehicle routing problem with stochastic demands
European J. Oper. Res.
(2009) A resilience-based framework for decision making based on simulation-optimization approach
Struct. Saf.
(2021)
Stochastic optimal control methodologies in risk-informed community resilience planning
Struct. Saf.
Optimal stochastic dynamic scheduling for managing community recovery from natural hazards
Reliab. Eng. Syst. Saf.
Near-optimal planning using approximate dynamic programming to enhance post-hazard community resilience management
Reliab. Eng. Syst. Saf.
Restoring infrastructure systems: An integrated network design and scheduling (INDS) problem
European J. Oper. Res.
A survey on the inventory-routing problem with stochastic lead times and demands
J. Appl. Log.
Solving the dynamic ambulance relocation and dispatching problem using approximate dynamic programming
European J. Oper. Res.
Risk-averse optimization of disaster relief facility location and vehicle routing under stochastic demand
Transp. Res. Part E
Network flows
A mathematical model for post-disaster road restoration: Enabling accessibility and evacuation
Transp. Res. Part E
Lateness minimization in pairwise connectivity restoration problems
INFORMS J. Comput.
Neuro-Dynamic Programming, Vol. 5
Dynamic Programming and Optimal Control, Vol. 1
A multi-objective robust stochastic programming model for disaster relief logistics under uncertainty
OR Spect.
A framework to quantitatively assess and enhance the seismic resilience of communities
Earthquake Spectra
Integrating restoration and scheduling decisions for disrupted interdependent infrastructure systems
Ann. Oper. Res.
The post-disaster debris clearance problem under incomplete information
Oper. Res.
In puerto rico, containers full of goods sit undistributed at ports
Vehicle routing with stochastic demands: Properties and solution frameworks
Transp. Sci.
Public assistance: Debris management guide
Debris estimating field guide
Hazus: Fema’s methodology for estimating potential losses from disasters
Earthquake model technical manual
Cited by (21)
A comprehensive approach in post-earthquake blockage prediction of urban road network and emergency resilience optimization
2024, Reliability Engineering and System SafetyA logic-based Benders decomposition algorithm for a repair crew routing and drone scheduling problem after a natural disaster
2023, Computers and Industrial EngineeringResource-based seismic resilience optimization of the blocked urban road network in emergency response phase considering uncertainties
2023, International Journal of Disaster Risk ReductionCitation Excerpt :OR/MS based studies commonly employ Graphs theory or network modeling and linear/dynamic programming techniques to introduce mathematical restoration models. Various objective functions have been used to optimize the recovery schedule such as network accessibility [34,35], connectivity [36,37], centrality [38], traffic flow and capacity, travel time and travel distance [39–41]. In these studies, it is usually assumed that a hypothetical time for path opening and network restoration following an earthquake [42–46].
Trends and gaps in the literature of road network repair and restoration in the context of disaster response operations
2022, Socio-Economic Planning SciencesCitation Excerpt :Aside from the uncertainty regarding which roads were damaged, they assume that communities demands are also stochastic. Similar to Aslan & Çelik [67]; Ulusan & Ergun [66] consider communities’ uncertain demands. In their problem, a set of critical vertices have an unmet demand and road clearing teams must improve the accessibility to these vertices.
Towards understanding network topology and robustness of logistics systems
2022, Communications in Transportation ResearchCitation Excerpt :From managerial perspective, our findings suggest that establishing regular services between distribution centers and/or building new distribution centers which allow bypass routes in the network are preferable. The robustness of a supply chain network is closely related to the concept of resilience (Ponomarov and Holcomb, 2009; Pettit et al., 2010; Tukamuhabwa et al., 2015; Ivanov, 2017; Hosseini et al., 2019; Dolgui et al., 2020; Chen and Chang, 2021; Ulusan and Ergun, 2021). Resilience is the capacity of an enterprise that enables it “to survive, adapt, and grow in the face of turbulent change” (Fiksel, 2006).
Beneficiary-centric decision support framework for enhanced resource coordination in humanitarian logistics: A case study from ASEAN
2022, Transportation Research Part E: Logistics and Transportation ReviewCitation Excerpt :Unlike business logistics, emergency logistics, i.e., humanitarian logistics or disaster response logistics, involves a unique set of challenges (Sheu, 2007). Unpredictable demand, shortage of essential supplies and services, disrupted infrastructures, short lead times, a chaotic operating environment, poor visibility of disaster information, spike in prices, and rapid changes in requirements on the ground are among the common challenges faced with humanitarian supply chain management (Lu et al. 2016; Sheu, 2010; Ulusan and Ergun, 2021; Wang et al., 2021b; Nagurney et al. 2016; Van Wassenhove, 2006; Eftekhar et al., 2014). In addressing these ever-increasing challenges, the current approach of adopting an organization- and execution-centric approach is no longer sufficient.