In this paper, we present a perception-action-communication loop design using Vision-based Graph Aggregation and Inference (VGAI). This multi-agent decentralized learning-to-control framework maps raw visual observations to agent actions, aided by local communication among neighboring agents. Our framework is implemented by a cascade of a convolutional and a graph neural network (CNN/GNN), addressing agent-level visual perception and feature learning, as well as swarm-level communication, local information aggregation and agent action inference, respectively. By jointly training the CNN and GNN, image features and communication messages are learned in conjunction to better address the specific task. We use imitation learning to train the VGAI controller in an offline phase, relying on a centralized expert controller. This results in a learned VGAI controller that can be deployed in a distributed manner for online execution. Additionally, the controller exhibits good scaling properties, with training in smaller teams and application in larger teams. Through a multi-agent flocking application, we demonstrate that VGAI yields performance comparable to or better than other decentralized controllers, using only the visual input modality and without accessing precise location or motion state information.

中文翻译：

---

具有卷积和图神经网络的可扩展感知-动作-通信循环

在本文中，我们提出了一种使用基于视觉的图聚合和推理 (VGAI) 的感知-动作-通信循环设计。这种多智能体分散式学习到控制框架将原始视觉观察映射到智能体动作，并辅以相邻智能体之间的本地通信。我们的框架由卷积和图神经网络 (CNN/GNN) 的级联实现，分别解决代理级视觉感知和特征学习，以及群级通信、局部信息聚合和代理动作推理。通过联合训练 CNN 和 GNN，联合学习图像特征和通信消息，以更好地解决特定任务。我们使用模仿学习在离线阶段训练 VGAI 控制器，依赖于集中式专家控制器。这导致学习的 VGAI 控制器可以以分布式方式部署以进行在线执行。此外，控制器表现出良好的扩展特性，在较小的团队中进行训练，在较大的团队中应用。通过多智能体植绒应用程序，我们证明了 VGAI 产生的性能与其他分散式控制器相当或更好，仅使用视觉输入模式，而无需访问精确的位置或运动状态信息。