In multi-agent reinforcement learning (MARL), information fusion through relationship modeling can effectively learn behavior strategies. However, the high dynamics among heterogeneous interactive agents in mixed cooperative–competitive environments pose difficulties for relational modeling. Traditional MARL solutions concatenate all agents’ states based on the global relationship, which is unrealistic and unscalable under large-scale conditions. Other methods fuse local information by modeling neighbor relationships, but local features lead to suboptimal strategies. From this perspective, we propose a novel relational information fusion approach for mixed tasks to fuse local and global features by modeling heterogeneous relationships through a hierarchical graph. During training, remote agents’ global features are fused through second-order graph representation to help strategy optimization. During decision making, the practicality and scalability of strategy are improved by fusing neighbor agents’ local features through first-order graph representation. Our approach consistently outperforms the state-of-the-art MARL methods in several multi-agent tasks, such as the Predator–Prey and Soccer Games. In particular, it achieves an 83.7% win rate, which is 11.5% higher than baselines in the 4 vs. 4 Soccer Game, and can scale from 4 vs. 4 to 13 vs. 13 in the Predator–Prey Game while maintaining good performance.

中文翻译：

---

混合合作竞争环境下多智能体强化学习的层次关系建模

在多智能体强化学习（MARL）中，通过关系建模进行信息融合可以有效地学习行为策略。然而，混合合作竞争环境中异构交互主体之间的高动态性给关系建模带来了困难。传统的MARL解决方案基于全局关系串联所有智能体的状态，这在大规模条件下是不现实且不可扩展的。其他方法通过建模邻居关系来融合局部信息，但局部特征会导致策略不理想。从这个角度来看，我们提出了一种用于混合任务的新型关系信息融合方法，通过层次图建模异构关系来融合局部和全局特征。在训练过程中，远程代理的全局特征通过二阶图表示融合，以帮助策略优化。在决策过程中，通过一阶图表示融合邻居代理的局部特征，提高了策略的实用性和可扩展性。我们的方法在多个多智能体任务中始终优于最先进的 MARL 方法，例如捕食者-猎物和足球游戏。特别是，它实现了 83.7% 的胜率，比 4 对 4 足球比赛的基准高出 11.5%，并且可以在 Predator-Prey 比赛中从 4 对 4 扩展到 13 对 13，同时保持良好的性能。