Learning multi-agent communication with double attentional deep reinforcement learning

Abstract

Communication is a critical factor for the big multi-agent world to stay organized and productive. Recently, Deep Reinforcement Learning (DRL) has been adopted to learn the communication among multiple intelligent agents. However, in terms of the DRL setting, the increasing number of communication messages introduces two problems: (1) there are usually some redundant messages; (2) even in the case that all messages are necessary, how to process a large number of messages in an efficient way remains a big challenge. In this paper, we propose a DRL method named Double Attentional Actor-Critic Message Processor (DAACMP) to jointly address these two problems. Specifically, DAACMP adopts two attention mechanisms. The first one is embedded in the actor part, such that it can select the important messages from all communication messages adaptively. The other one is embedded in the critic part so that all important messages can be processed efficiently. We evaluate DAACMP on three multi-agent tasks with seven different settings. Results show that DAACMP not only outperforms several state-of-the-art methods but also achieves better scalability in all tasks. Furthermore, we conduct experiments to reveal some insights about the proposed attention mechanisms and the learned policies.

Appendices

Appendix 1: The hyperparameters

See Tables 3, 4 and 5.

Table 3 The hyperparameters used in cooperative navigation tasks

Table 4 The hyperparameters used in packet routing tasks

Table 5 The hyperparameters used in traffic control tasks

Appendix 2: The layer merging method

This section introduces the layer-merging method that transforms the multi-dimensional action conditional Q-values into scalar Q-values.

Originally, we want to implement the following equation (here, we take $K=3$ as an example):

$$\begin{aligned} Q_i = w_1 Q_{i1} + w_2 Q_{i2} + w_3 Q_{i3} \end{aligned}$$

(13)

where $Q_i$ is the real Q-value used in Bellman equations, $Q_{ik}$ is the k-th Q-value head, and $w_k$ is the weight of each Q-value head, respectively. Note that in the equation, $Q_{ik}$ is a scalar. This is the original naive idea.

However, as mentioned in the main paper, in our real implementation, the action conditional Q-values$Q^{k}_{i}$ (i.e., the Q-value heads) and the contextual Q-value$Q^{c}_{i}$ are 32D vectors that mimic the scalar Q-value. To generate the real Q-value$Q_{i}$ used in Bellman equations, we further add a fully-connected layer, which has one output node representing the real Q-value$Q_{i}$, after the contextual Q-value$Q^{c}_{i}$.

With the above precondition, we use the variables in the main paper (i.e., in our real implementation) to calculate$\varvec{w}_{\varvec{k}}$and$\varvec{Q}_{\varvec{ik}}$ (and accordingly, $\varvec{Q}_{\varvec{i}}$) in Eq. (13) in a suitable way.

Recall that, in the real implementation, $Q_i$ is generated using:

$$\begin{aligned} Q_i = l^1 {Q^{c}_{i}}^1 + l^2 {Q^{c}_{i}}^2 + \cdots + l^{32} {Q^{c}_{i}}^{32} \end{aligned}$$

(14)

where $Q^{c}_{i}$ is the 32D contextual Q-value, ${Q^{c}_{i}}^{m}$ is the m-th element of $Q^{c}_{i}$, and $l^m$ is the m-th network weight which links ${Q^{c}_{i}}^{m}$ and $Q_i$. Note that $l^m$ is a scalar, and the last layer of the critic network can be denoted as $L=[l^1, l^2, \ldots , l^{32}]$

Recall that, in the real implementation, the contextual Q-value$Q^{c}_{i}$ is generated using:

$$\begin{aligned} Q^{c}_{i} = W^1 Q^{1}_{i} + W^2 Q^{2}_{i} + W^3 Q^{3}_{i} \end{aligned}$$

(15)

where $Q^{k}_{i}$ is the action conditional Q-values (each of which is a 32D vector), and $W=\langle W^1, W^2, W^3 \rangle$ is the learned attention weight where $W^1 + W^2 + W^3 = 1$. Note that $W^k$ is a scalar. Accordingly, the m-th element of $Q^{c}_{i}$ is generated using:

$$\begin{aligned} {Q^{c}_{i}}^{m} = W^1 {[Q^{1}_{i}]}^{m} + W^2 {[Q^{2}_{i}]}^{m} + W^3 {[Q^{3}_{i}]}^{m} \end{aligned}$$

(16)

Then we can rewrite Eq. (14) as:

$$\begin{aligned} Q_i&= l^1 {Q^{c}_{i}}^1 + l^2 {Q^{c}_{i}}^2 + \cdots + l^{32} {Q^{c}_{i}}^{32} \nonumber \\&= l^1(W^1 {[Q^{1}_{i}]}^{1} + W^2 {[Q^{2}_{i}]}^{1} + W^3 {[Q^{3}_{i}]}^{1}) \cdots \nonumber \\&\quad + l^{32}(W^1 {[Q^{1}_{i}]}^{32} + W^2 {[Q^{2}_{i}]}^{32} + W^3 {[Q^{3}_{i}]}^{32}) \end{aligned}$$

(17)

$$\begin{aligned}&= W^1(l^1 {[Q^{1}_{i}]}^{1} + l^2 {[Q^{1}_{i}]}^{2} + \cdots + l^{32} {[Q^{1}_{i}]}^{32}) \nonumber \\&\quad + W^2(l^1 {[Q^{2}_{i}]}^{1} + l^2 {[Q^{2}_{i}]}^{2} + \cdots + l^{32} {[Q^{2}_{i}]}^{32}) \nonumber \\&\quad + W^3(l^1 {[Q^{3}_{i}]}^{1} + l^2 {[Q^{3}_{i}]}^{2} + \cdots + l^{32} {[Q^{3}_{i}]}^{32}) \end{aligned}$$

(18)

Comparing Eq. (18) with Eq. (13), we can calculate $w_k$ and $Q_{ik}$ using:

$$\begin{aligned} w_1 &= W^1 \end{aligned}$$

(19)

$$\begin{aligned} w_2 &= W^2 \end{aligned}$$

(20)

$$\begin{aligned} w_3 &= W^3 \end{aligned}$$

(21)

$$\begin{aligned} Q_{i1} &= l^1 {[Q^{1}_{i}]}^{1} + l^2 {[Q^{1}_{i}]}^{2} + \cdots + l^{32} {[Q^{1}_{i}]}^{32} \end{aligned}$$

(22)

$$\begin{aligned} Q_{i2} &= l^1 {[Q^{2}_{i}]}^{1} + l^2 {[Q^{2}_{i}]}^{2} + \cdots + l^{32} {[Q^{2}_{i}]}^{32} \end{aligned}$$

(23)

$$\begin{aligned} Q_{i3} &= l^1 {[Q^{3}_{i}]}^{1} + l^2 {[Q^{3}_{i}]}^{2} + \cdots + l^{32} {[Q^{3}_{i}]}^{32} \end{aligned}$$

(24)

As can be seen from the above equations, this method directly connects the action conditional Q-values$Q^k_i$ with the last layer of the critic network L to transform the multi-dimensional Q-value into scalar Q-value. It can be seen as a layer merging method. We expect that the above analysis is acceptable.