Real-time deep reinforcement learning based vehicle navigation

Highlights

• A novel deep reinforcement learning method for real-time vehicle navigation.

• Smart agents are embedded into SUMO simulator in a dynamic transportation system.

• Performance is validated by nine realistic road and traffic combined conditions.

Abstract

Traffic congestion has become one of the most serious contemporary city issues as it leads to unnecessary high energy consumption, air pollution and extra traveling time. During the past decade, many optimization algorithms have been designed to achieve the optimal usage of existing roadway capacity in cities to leverage the problem. However, it is still a challenging task for the vehicles to interact with the complex city environment in a real time manner. In this paper, we propose a deep reinforcement learning (DRL) method to build a real-time intelligent vehicle routing and navigation system by formulating the task as a sequence of decisions. In addition, an integrated framework is provided to facilitate the intelligent vehicle navigation research by embedding smart agents into the SUMO simulator. Nine realistic traffic scenarios are simulated to test the proposed navigation method. The experimental results have demonstrated the efficient convergence of the vehicle navigation agents and their effectiveness to make optimal decisions under the volatile traffic conditions. The results also show that the proposed method provides a better navigation solution comparing to the benchmark routing optimization algorithms. The performance has been further validated by using the Wilcoxon test. It is found that the achieved improvement of our proposed method becomes more significant under the maps with more edges (roads) and more complicated traffics comparing to the state-of-the-art navigation methods.

Introduction

In recent years, traffic congestion in urban area has become a serious problem due to the rapid development of urbanization. It brings a major impact on urban transportation networks that lead to extra traveling hours, increased fuel consumption and air pollution. According to the study in [1], traffic congestion can be categorized into recurring congestion (RC) and non-recurring congestion (NRC). NRC is defined as the congestion made by unexpected events, such as construction work, inclement weather, accidents, and special events [2]. Unsurprisingly, NRC accounts for a larger proportion of traffic delays in urban areas comparing to the RC due to its unpredictable nature [3]. There are three categories of methods proposed to tackle the NRC problem: (1) detecting and predicting traffic congestions by utilizing both the historical and real-time sensor data [4], [5]; (2) optimizing traffic signal control and management [6], [7]; and (3) vehicle routing and navigation optimization [8], [9], [10], [11]. Wherein, the vehicle routing and navigation, as the most promising solution, has been investigated extensively during the past decades.

Classical vehicle routing problem (VRP) is defined as finding the minimum cost of the combined routes of a given number of vehicles m to serve n customers. One of the typical examples is path planning for collecting and sending packages from a delivery company. Traditional VRP is formed and classified as an NP-hard problem [9]. Many optimization algorithms are proposed to find the sub-optimal solution under different constraints, e.g. genetic algorithm [12], firefly algorithm [13], hybrid algorithm [14] or backbone algorithm [15]. However, this type of routing problem definition assumes a relatively stable traffic condition, and target on the travel salesman problem. Thus, these proposed optimization algorithms search for optimal solution under a static context. Different from the classical VRP, this study targets the vehicle routing and navigation problem as controlling the vehicle to make path planning from a given start node to a destination node with shortest travel time under dynamic traffic conditions [16]. The objective is to find a path or a set of optimal actions to minimize travel time with real-time collected input of current traffic conditions. In the past few years, many heuristic and evolutionary optimization algorithms are proposed to solve the problem [17], [18], [19], [20], [21]. Although recent research on vehicle routing and navigation has achieved reasonable results, the state-of-the-art methods suffer from several drawbacks: firstly, the shortest path type of methods, e.g., [16], [18], [21], becomes less efficient to provide optimal solutions due to unpredictable nature of more complicated traffic conditions and cannot react instantly to NRC issues. Secondly, it is hard to solve the problem in a real-time manner when using optimization algorithms, e.g., [20], [22], [23]. The searching space grows exponentially when more routing edges are added into the map. Thirdly, optimization based vehicle routing and navigation algorithms, such as [19], [24], [25], [26], cannot perform self-evolution and self-adaptation. To address the limitations of the methods, this paper proposes a deep reinforcement learning (DRL) method to achieve real-time intelligent vehicle navigation to alleviate the NRC issues.

By formulating vehicle routing and navigation task as a sequence of decisions, the DRL framework can be utilized to solve this automated control problem through taking real-time observations and evaluating the outcomes from actions under a complex environment. Inspired by recent success of deep reinforcement learning methods, the enhanced deep-Q network (DQN) method is adopted to deal with this real-world complexity given the fact that navigation task can be modeled as a Markov Decision Process (MDP). Comparing to the DQN in [27], our method enhances it in three aspects: (1) a suitable reward function is designed for the vehicle routing and navigation task; (2) several advanced schemes, including double DQN, dueling DQN and priority experience replay, are integrated into the framework to achieve a more reliable convergence of the network; and (3) a distance ranking sampling strategy is proposed to speed up the convergence speed.

In our work, a traffic simulator, SUMO, is seamlessly connected to smart navigation agents to provide an integrated experimental framework. It could simulate real-world traffic conditions as well as embed agent decisions into the traffic simulator. Once agents are required to make navigation decisions, observations generated from simulated traffic environment are fed into these agents as current traffic state representations. The agents could, therefore, make automated real-time navigation decisions that minimize the travel time to reach their destination. The proposed DRL framework provides an appealing and innovative solution for vehicle routing and navigation problem as the learning process is fully automated which does not require any labeling or guidance. Furthermore, the agents can automatically adapt their policy networks by analyzing global environment as their context and eventually decrease travel time when new data is generated.

The main contributions of our research can be highlighted as follows: (1) this work proposes a novel DRL algorithm to achieve an effective real-time vehicle routing and navigation system; (2) the DRL agents are embedded into traffic simulator SUMO to achieve an integrated framework to facilitate the intelligent vehicle navigation research under the context of a dynamic urban transportation system; and (3) the potential practical usage is demonstrated in nine realistic road and traffic combined conditions. Further, its efficiency is validated by using the Wilcoxon test.

The remainder of the paper is organized as follows: The background of our research is described in Section 2 to clarify the motivation of our work. Section 3 provides an overview of the proposed framework and introduces the main components in our system. Section 4 and Section 5 explain the detail of two main components, i.e. the traffic simulator and DRL smart agents. In Section 6, the experimental results are presented to demonstrate the convergence of the DRL agents, how the agents navigate vehicles to make routing choice and their performance compared to the SUMO build-in route optimization algorithms. Finally, Section 7 presents the conclusive remarks and suggests future work.