当前位置: X-MOL 学术arXiv.cs.GT › 论文详情
Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
arXiv - CS - Computer Science and Game Theory Pub Date : 2020-02-17 , DOI: arxiv-2002.07066
Qiaomin Xie; Yudong Chen; Zhaoran Wang; Zhuoran Yang

We develop provably efficient reinforcement learning algorithms for two-player zero-sum Markov games in which the two players simultaneously take actions. To incorporate function approximation, we consider a family of Markov games where the reward function and transition kernel possess a linear structure. Both the offline and online settings of the problems are considered. In the offline setting, we control both players and the goal is to find the Nash Equilibrium efficiently by minimizing the worst-case duality gap. In the online setting, we control a single player to play against an arbitrary opponent and the goal is to minimize the regret. For both settings, we propose an optimistic variant of the least-squares minimax value iteration algorithm. We show that our algorithm is computationally efficient and provably achieves an $\tilde O(\sqrt{d^3 H^3 T})$ upper bound on the duality gap and regret, without requiring additional assumptions on the sampling model. We highlight that our setting requires overcoming several new challenges that are absent in Markov decision processes or turn-based Markov games. In particular, to achieve optimism in simultaneous-move Marko games, we construct both upper and lower confidence bounds of the value function, and then compute the optimistic policy by solving a general-sum matrix game with these bounds as the payoff matrices. As finding the Nash Equilibrium of such a general-sum game is computationally hard, our algorithm instead solves for a Coarse Correlated Equilibrium (CCE), which can be obtained efficiently via linear programming. To our best knowledge, such a CCE-based scheme for implementing optimism has not appeared in the literature and might be of interest in its own right.
更新日期:2020-03-24

 

全部期刊列表>>
欢迎访问IOP中国网站
自然职场线上招聘会
GIANT
产业、创新与基础设施
自然科研线上培训服务
材料学研究精选
胸腔和胸部成像专题
屿渡论文,编辑服务
何川
苏昭铭
陈刚
姜涛
李闯创
李刚
北大
跟Nature、Science文章学绘图
隐藏1h前已浏览文章
中洪博元
课题组网站
新版X-MOL期刊搜索和高级搜索功能介绍
ACS材料视界
x-mol收录
上海纽约大学
张健
陈芬儿
厦门大学
李祥
吉林大学
卓春祥
张昊
杨中悦
试剂库存
天合科研
down
wechat
bug