A composite learning method for multi-ship collision avoidance based on reinforcement learning and inverse control

Abstract

Model-free reinforcement learning methods have potentials in ship collision avoidance under unknown environments. To defect the low efficiency problem of the model-free reinforcement learning, a composite learning method is proposed based on an asynchronous advantage actor-critic (A3C) algorithm, a long short-term memory neural network (LSTM) and Q-learning. The proposed method uses Q-learning for adaptive decisions between a LSTM inverse model-based controller and the model-free A3C policy. Multi-ship collision avoidance simulations are conducted to verify the effectiveness of the model-free A3C method, the proposed inverse model-based method and the composite learning method. The simulation results indicate that the proposed composite learning based ship collision avoidance method outperforms the A3C learning method and a traditional optimization-based method.

Introduction

The automation of vehicles is an effective approach to reduce the risk caused by dangerous driving behaviors [20], [71]. Due to the safety requirements of the autonomous surface vessels (ASVs), a collision avoidance system becomes more and more important [4]. With the recent development of artificial intelligence, deep reinforcement learning (DRL) methods [34], [44] have profound effects on complex decision-making tasks like multi-ship collision avoidance, for which the DRL method provides an alternative approach compared with the traditional methods. As a branch of DRL methods, the policy gradient-based methods (e.g., deep deterministic policy gradient (DDPG) [25], asynchronous advantage actor-critic (A3C) [33], proximal policy optimization (PPO) [41], etc) can solve continuous action problems with a deterministic policy parameterized by neural networks. At present, the DRL has made breakthroughs in a variety of domains, such as control [38], path planning [29], collision avoidance [10] and so on. In this section, the existing ship collision avoidance methods are introduced, including the traditional methods and DRL based methods.

Traditional ship collision avoidance methods can be briefly divided to two categories: path generation methods and intelligent optimization methods.

Path generation methods mainly include global grid based methods (e.g., A* [47], [26]) and local path generation methods (e.g., artificial potential field (APF) [62], [63]). As a heuristic search algorithm, A* considers both the origin and the destination, which has a global optimality. Hierarchical planning [53], [8] and additional constraints [11], [48] are the commonly used approaches in A* to improve the search efficiency and path smoothness. Compared with A*, APF [31], [24] has smaller computation and smoother paths by using artificial gravitational and repulsive field to model the navigation environment.

With the development of intelligent optimization algorithms, the optimization based methods, e.g., fuzzy mathematics, neural networks and swarm intelligence, have attracted more attention in ship collision avoidance [49] than before. Fuzzy mathematics has been applied in the fuzzy classification [13], [12] and reasoning [39], [40] of ship collision risk for a long time. Generally, the output of fuzzy mathematics relies on the membership functions set in advance, which needs more prior knowledge. Besides, neural networks [17] is another powerful approach to model the uncertain factors in reasoning the ship collision risk. The neural networks are commonly combined with fuzzy mathematics [2] and expert system (ES) [46] to realize reliable collision avoidance.

Recently, swarm intelligence and evolution optimization methods have become hot topics in ship collision avoidance. Ant colony optimization (ACO) [22], [23], [51] and particle swarm optimization (PSO) [32], [35], [7], [28] are the most commonly used algorithms in ship collision avoidance, which can obtain good results with an appropriate fitness function.

At present, several DRL methods have been applied in ship collision avoidance. The key issues in the DRL methods are the definitions of the state, action and reward function.

Deep Q-network is firstly applied in ship collision avoidance with a low-dimensional state-action space. In Ref. [42], the discrete ship heading changes and a set of distances measured by the fixed interval detection lines around the ship are defined as the actions and states in deep Q-network, respectively. The proposed deep Q-network is verified in both simulations [42] and model ship experiments [43]. In Ref. [65], a so-called constrained deep Q-network is proposed to reduce the complexity of the state-action space by adding constrains based on international regulations for preventing collisions at sea (COLREGs), which also obtains good collision avoidance results.

In addition, using high-complex deep neural networks is an effective approach to train the value function in a high-dimensional state-action space. In Ref. [9], a convolutional neural network (CNN) is used to obtain a more reliable policy. This CNN model uses the perception information, the motion state and the actions of the ship in a certain horizon to produce a chain lumped state matrix for reinforcement learning.

Although deep Q-network based methods have achieved good results, a large memory space is still required for the discrete actions. With the development of policy gradient-based methods (e.g., DDPG, A3C, etc), a deterministic policy from ship states to continuous actions can be obtained with less memory space. In Ref. [19] the DDPG algorithm is applied in a simplified state-action space, in which the vertical distances away from the target course are defined as the continuous actions. In Xu et al. [59], the DDPG algorithm is also applied for ship collision avoidance with the same state-action space defined in Cheng and Zhang [9]. The simulation results have indicated the effectiveness and advantages of DDPG algorithm in continuous decisions for ship collision avoidance.

Although existing DRL methods have obtained rich achievements in ship collision avoidance, the relative low efficiency is the main barrier in application. At present, the asynchronous computing framework (e.g., A3C) and model-based for model-free (i.e., MB-MF) learning can both improve the learning efficiency. To reduce the exploring time and memory costs, an on-policy A3C method [33] updates the obtained gradients through asynchronous parallel computing, which has greater potentials than the experience reply technique [37] in DDPG. For MB-MF learning methods, the main idea is to establish a short-term model-based optimizer for combination with the long-term learner. In Ref. [5], a linear-quadratic regulator (LQR) optimizer is adopted for data-efficient learning in both simulations and real-world experiments. This LQR optimizer uses a time-varying linear-Gaussian (TVLG) model for optimization, which is established by fitting the samples. In Ref. [36], the widely used model predictive control (MPC) approach [70], [67], [68], [69] is applied for model-based learning, which can initialize the networks for accelerating the model-free learning. The model used in MPC is a dynamic model that predicts the state changes over the time step duration.

Surveying from the ship collision avoidance research, DRL methods obtain an optimal policy by maximizing future rewards through interactions, which have better potentials than the traditional methods in uncertain environments. In spite of this, the main drawback in existing DRL based collision avoidance methods is the low learning efficiency problem, especially the off-line methods such as DDPG.

As denoted in the related DRL works, the asynchronous computing and MB-MF learning can be considered to improve the learning efficiency for ship collision avoidance. Motivated by Refs. [33], [5], [36], the main works of this paper are as follows:

• To reduce the learning time and memory costs, the A3C method [33] is applied for ship collision avoidance in this study.

• The MB-MF learning is considered to further improve the learning efficiency. Besides, traditional model-based method [5], [36] needs to establish an accurate model before designing the controller. While the inverse control [55], [3], [14] method directly uses the inverse model between the desired outputs and inputs as the controller, which has more concise strategy and potentials in MB-MF learning.

In order to improve the learning efficiency for ship collision avoidance, a composite learning method is proposed in this study. A simple framework of the proposed composite learning is shown in Fig. 1 for brief explanation. Instead of the simple initialization in Ref. [36], the main idea of this method is to use Q-learning for adaptive decisions between the actions of a model-free A3C and a model-based inverse controller in the entire learning process. The contributions of this study are:

• Compared with traditional ship collision avoidance methods, the proposed method has efficient learning ability by integrating A3C reinforcement learning, which performs better with limited perception and states.

• Compared with pure model-free A3C method, the proposed composite method uses a feed-forward LSTM controller and Q-learning to generate supervised trajectories for A3C, which has higher learning efficiency.

Therefore, in addition to A3C applications, the originality of the proposed composite learning method is reflected in two aspects: the inverse controller for ship collision avoidance and decisions based on Q-learning.

The remainder of this article is organized as follows. In Section 2, the ship hydrodynamic model and collision risk model are described. In Section 3, the A3C learning based ship collision avoidance method is described. In Section 4, the composite learning method is proposed. In Section 5, simulation experiments under multi-ship encounters are carried out to assess the effectiveness of the proposed methods. In Section 6, conclusions and future research are presented.