The impact of agent definitions and interactions on multiagent learning for coordination in traffic management domains

Abstract

The state-action space of an individual agent in a multiagent team fundamentally dictates how the individual interacts with the rest of the team. Thus, how an agent is defined in the context of its domain has a significant effect on team performance when learning to coordinate. In this work we explore the trade-offs associated with these design choices, for example, having fewer agents in the team that individually are able to process and act on a wider scope of information about the world versus a larger team of agents where each agent observes and acts in a more local region of the domain. We focus our study on a traffic management domain and highlight the trends in learning performance when applying different agent definitions. In addition, we analyze the impact of agent failure for different agent definitions and investigate the ability of the team to learn new coordination strategies when individual agents become unresponsive.

Appendix: Additional experiments on the centralized agent network architecture

We conducted additional experiments to assess the performance of the centralized learner when a more complex network architecture was available. In these experiments, the centralized agent uses the same relative number of hidden neurons as the intersection agents, i.e. four times the number of edges, which for 38 edges gives 152 hidden neurons. Thus, the centralized$_{\text {rel}}$ agent has 11,590 weights defining its control policy while the centralized$_{\text{t,rel}}$ agent with time information has 17,366 weights. This is over nine times more parameters than the originally tested centralized and centralized$_{\text {t}}$ agent policies (see Table 1). Results are shown in Figs. 10, 11 and 12 below where the original centralized policies are in green and cyan (no time information and with time information, respectively), while the performance of the larger neural network policies are shown in dark grey and light grey, respectively.

Fig. 10

Comparison of the average team performance across training epochs for different centralized agent policy architectures. Green and cyan plots are equivalent to those shown in Figs. 4, 5 and 6, while the grey curves show the performance of centralized agents whose neural network control policies have over nine times more parameters. Mean and one standard deviation (from 30 statistical runs) are shown for each set of experiments with increasing numbers of AGVs from (a)–(d). Best viewed in color (Color figure online)

Fig. 11

Best team performance across training epochs. Best viewed in color (Color figure online)

From these results, we see that, in practice, additional policy parameters do not enable substantially better learning performance. The centralized learner with time information is able to improve its learning performance when using the more complex network architecture; however, the centralized learner without time information actually degrades in its performance when more parameters are introduced. Of course, a more comprehensive sweep of possible network architectures would be needed to assess whether there exists a “sweet spot” that balances network representational capacity and learning complexity for each of these cases. Nevertheless, the basic centralized learner with no time information and only 1254 network weights still outperforms all other policy architectures under all computed metrics. A quantitative comparison of the mean and medians of the final distributions is given in Table 4.

Table 4 Percentage improvement of the basic centralized agent against all other centralized policies

Finally, we note that in these experiments, we only run our learning algorithm for 500 learning epochs. However, the learning trends suggest that to achieve substantial improvements in the policy performance (e.g. beyond what is currently achieved by the intersection, intersection$_{\text {t}}$ and link$_{\text {t}}$ agents) will require a significantly greater learning effort. The bottleneck may lie in the underlying evolutionary algorithm that is used to train the policies. Thus, future work may consider adapting or replacing this routine with techniques that are tailored to efficiently training neural networks with large numbers of parameters.

Fig. 12

Violin plots of the team performance distributions at the end of 500 training epochs. The ‘+’ symbol represents the mean and the ‘$\times$’ represents the median. The centralized$_{\text {t,rel}}$ agent with 17,366 weights is able to improve on the performance of the centralized$_{\text {t}}$ agent, which only has 1862 weights. However, the basic centralized agent with only 1254 neural network weights produces the highest mean and median performance under all four tested domain complexities