Robotic agents must adopt existing social conventions in order to be effective teammates. These social conventions, such as driving on the right or left side of the road, are arbitrary choices among optimal policies, but all agents on a successful team must use the same convention. Prior work has identified a method of combining self-play with paired input-output data gathered from existing agents in order to learn their social convention without interacting with them. We build upon this work by introducing a technique called Adversarial Self-Play (ASP) that uses adversarial training to shape the space of possible learned policies and substantially improves learning efficiency. ASP only requires the addition of unpaired data: a dataset of outputs produced by the social convention without associated inputs. Theoretical analysis reveals how ASP shapes the policy space and the circumstances (when behaviors are clustered or exhibit some other structure) under which it offers the greatest benefits. Empirical results across three domains confirm ASP's advantages: it produces models that more closely match the desired social convention when given as few as two paired datapoints.

中文翻译：

---

对抗性引导的自我游戏以适应社会习俗

机器人代理必须采用现有的社会习俗才能成为有效的队友。这些社会惯例，例如在道路的右侧或左侧行驶，是最优策略中的任意选择，但成功团队中的所有代理都必须使用相同的惯例。先前的工作已经确定了一种将自我游戏与从现有代理收集的成对输入-输出数据相结合的方法，以便在不与他们互动的情况下学习他们的社会习俗。我们在这项工作的基础上引入了一种称为对抗性自我游戏 (ASP) 的技术，该技术使用对抗性训练来塑造可能的学习策略的空间并显着提高学习效率。ASP 只需要添加未配对的数据：由社会惯例产生的输出数据集，没有相关的输入。理论分析揭示了 ASP 如何塑造政策空间和环境（当行为聚集或表现出其他结构时），在这种情况下它可以提供最大的好处。三个领域的实证结果证实了 ASP 的优势：当给定两个配对的数据点时，它生成的模型更符合所需的社会习俗。