The combination of Formal Methods with Reinforcement Learning (RL) has recently attracted interest as a way for single-agent RL to learn multiple-task specifications. In this paper we extend this convergence to multi-agent settings and formally define Extended Markov Games as a general mathematical model that allows multiple RL agents to concurrently learn various non-Markovian specifications. To introduce this new model we provide formal definitions and proofs as well as empirical tests of RL algorithms running on this framework. Specifically, we use our model to train two different logic-based multi-agent RL algorithms to solve diverse settings of non-Markovian co-safe LTL specifications.

中文翻译：

---

扩展马尔可夫游戏以学习多智能体强化学习中的多个任务

形式方法与强化学习 (RL) 的结合最近引起了人们的兴趣，作为单代理 RL 学习多任务规范的一种方式。在本文中，我们将这种收敛性扩展到多智能体设置，并将扩展马尔可夫博弈正式定义为允许多个 RL 智能体同时学习各种非马尔可夫规范的通用数学模型。为了介绍这个新模型，我们提供了在这个框架上运行的 RL 算法的正式定义和证明以及经验测试。具体来说，我们使用我们的模型来训练两种不同的基于逻辑的多智能体 RL 算法，以解决非马尔可夫协同安全 LTL 规范的不同设置。