A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic en route sector. The concept of using distributed vehicle autonomy to ensure separation is proposed, instead of a centralized sector air traffic controller. Our proposed framework uses proximal policy optimization that is customized to incorporate an attention network. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. Agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents. The proposed framework is validated on three case studies in the BlueSky air traffic simulator. Several baselines are introduced, and the numerical results show that the proposed framework significantly reduces offline training time, increases safe separation performance, and results in a more efficient policy.

提出了一种新颖的深度多智能体强化学习框架，以识别和解决高密度、随机和动态航路扇区中可变数量飞机之间的冲突。提出了使用分布式车辆自治来确保分离的概念，而不是集中的扇区空中交通管制员。我们提出的框架使用近端策略优化，该优化被定制为包含一个注意力网络。这使代理能够以可扩展的、有效的方法访问该扇区中可变的飞机信息，以在不确定的情况下实现高交通吞吐量。代理使用集中学习、分散执行方案进行训练，其中一个神经网络由所有代理学习和共享。提议的框架在 BlueSky 空中交通模拟器中的三个案例研究中得到验证。引入了几个基线，数值结果表明，所提出的框架显着减少了离线训练时间，提高了安全分离性能，并产生了更有效的策略。