A differentiable path-following method to compute subgame perfect equilibria in stationary strategies in robust stochastic games and its applications

Highlights

• Compute subgame perfect equilibria in stationary strategies in stochastic games.

• Develop a logarithmic-barrier differentiable path-following method.

• Attain a convex-quadratic-penalty differentiable path-following method.

• Apply the proposed method to solve the problems on medical waste recycling.

Abstract

As an effective paradigm to address uncertainty in payoffs and transition probabilities, robust stochastic games have been formulated in the literature. This paper is concerned with the computation of subgame perfect equilibria in stationary strategies (SSPEs) in robust stochastic games. To tackle this problem, we develop in this paper a globally convergent differentiable path-following method by exploiting the structures of the games. Incorporating a logarithmic-barrier term into each player’s payoff function with an extra variable between zero and one, we constitute a logarithmic-barrier robust stochastic game in which each player solves in each state a convex optimization problem. An application of the optimality conditions to the barrier game together with a fixed-point argument yields a polynomial equilibrium system for the barrier game. As a result of this system, we establish the existence of a smooth path that starts from an arbitrary mixed strategy profile and ends at an SSPE as the extra variable descends from one to zero. As an alternative scheme, we make up a convex-quadratic-penalty robust stochastic game and attain a globally convergent convex-quadratic-penalty differentiable path-following method for SSPEs in robust stochastic games. Numerical comparisons show that the logarithmic-barrier path-following method significantly outperforms the convex-quadratic-penalty path-following method. To further evince the value of the proposed methods, we apply the logarithmic-barrier path-following method to solve a supply chain configuration problem and a market entry problem from medical waste recycling.

Introduction

Stochastic games model situations where competing players take actions over time to achieve their individual objectives. At each stage, the players choose their actions simultaneously and independently of each other and receive instantaneous payoffs. The game then moves into a next state according to a transition probability distribution, and continues thereon. Stochastic games were first formulated by Shapley (1953) with only two players. The extensions to multi-players were carried out by Fink (1964) and Takahashi (1964). Stochastic games have been widely applied in economic analysis. Ericson & Pakes (1995) analyze the behavior of individual firms in an evolving market place. Flesch, Thuijsman, & Vrieze (2007) deal with stochastic games in which the transition probabilities for an action profile in the current state can be decomposed into player-dependent components. Fershtman & Pakes (2012) develop a framework for empirical work of stochastic games and employ a heuristic iterative procedure to compute equilibrium strategies of oligopolies. Parilina, Sedakov, & Zaccour (2017) consider a stochastic dynamic game to determine the price of anarchy. Mandel & Venel (2020) formulate the problem of dynamic competition over social networks as a stochastic game. Garrec & Scarsini (2020) apply stochastic games to searching for an immobile hider on a stochastic network. Solan & Vieille (2015) summarize the historical developments of stochastic games and emphasize the importance of the seminal work in Shapley (1953). To address uncertainty on payoffs and transition probabilities, Kardeş, Ordóñez, & Hall (2011) incorporate robust optimization into a finite discounted stochastic game and formulate a robust stochastic game, which can be regarded as an extension of robust normal-form games introduced by Aghassi & Bertsimas (2006). An application of robust games to a decision problem is presented in Caballero, Lunday, & Uber (2021) where players are uncertain of the opponents’ reasoning abilities. Liu, Xu, Yang, & Zhang (2018) develop several distributionally robust equilibrium models in which players lack complete information on the true probability distribution of uncertainty. Shapiro (2021) extends distributionally robust approaches to multistage stochastic programming. Other applications of robust games can be found in Jiang, Netessine, & Savin (2011) and Zhu, Zhang, & Ye (2013). In robust games, each player accepts a robust optimization approach towards the uncertainty and intends to optimize the worst-case performance, where the worst-case occurs over the ambiguity sets for uncertain parameters.

This paper is concerned with the computation of subgame perfect equilibria in stationary strategies (SSPEs) in robust stochastic games. The concept of SSPE was characterized in Maskin & Tirole (2001), which plays a significant role in stochastic games as demonstrated by Adlakha, Johari, & Weintraub (2015). A subgame perfect equilibrium in stationary strategies depends only on the current state. As mentioned in Herings & Peeters (2004), a stationary strategy profile is an SSPE if each player learns to response optimally in all states. There has been much interest in the computation of SSPEs in stochastic games. Filar, Schultz, Thuijsman, & Vrieze (1991) convert SSPEs in finite discounted stochastic games as global optima of certain nonlinear programs. Herings & Peeters (2004) develop a stochastic tracing procedure to compute SSPEs in stochastic games, where one needs to apply an iterative method several times to find a starting point. To mitigate this deficiency, Li & Dang (2020) present an arbitrary starting stochastic tracing procedure. Borkovsky, Doraszelski, & Kryukov (2010) give a guide to computing SSPEs in stochastic games by a general differentiable homotopy method. Although these methods have significantly advanced the applications of stochastic games, they cannot be directly applied for computing SSPEs in robust stochastic games.

Since an SSPE is a Nash equilibrium in stationary strategies, the computation of SSPEs is closely related to the computation of Nash equilibria. As a class of the most effective mechanisms for computing Nash equilibria, globally convergent path-following methods have been developed in the literature. The first path-following method was described in the seminal work of Lemke & Howson (1964) to compute a Nash equilibrium of a two-person game. The existence of Nash equilibrium was established by Nash (1951) through an equivalent fixed point problem. To compute fixed points of continuous mappings, simplicial path-following methods were pioneered by Scarf (1967) and substantially developed in the literature such as Allgower & Georg (2003), Dang (1991), Eaves (1972), Eaves & Saigal (1972), Kojima & Yamamoto (1984), van der Laan & Talman (1979), Scarf (1973) and Todd (1976). To enjoy differentiability of a mapping, differentiable path-following methods were invented by Kellogg, Li, & Yorke (1976) through a constructive proof to Brouwer’s fixed-point theorem. Some further developments of differentiable path-following methods can be found in Garcia & Zangwill (1981), Kubler, Renner, & Schmedders (2014) and Watson (2001) and the references therein. Nash equilibria can also be reformulated as solutions to a variational inequality problem or a stationary point problem of a smooth mapping on a polytope. Simplicial path-following methods are expanded to computing a stationary point in Dai, van der Laan, Talman, & Yamamoto (1991). Differentiable path-following methods are developed for computing stationary points in the literature such as Ding (1993), Hale, Yin, & Zhang (2008), Shang, Xu, & Yu (2011) and Zhou & Yu (2014). Recently, Migot & Cojocaru (2020) propose a parameterized variational inequality scheme to track the solution set for a generalized Nash equilibrium problem. To take advantage of special structures of games, several path-following methods have been specifically devised for computing Nash equilibria in the literature. By subdividing the product of strategy spaces into simplices, simplicial path-following methods have been tailored for computing Nash equilibria in Doup & Talman (1987), van den Elzen & Talman (1999), Govindan & Wilson (2010) and von Stengel, van den Elzen, & Talman (2002). Integrating a linear term into the payoff of each player, Herings & Peeters (2001) derive a differentiable path-following method to select a Nash equilibrium. Utilizing a structure theorem for the Nash equilibrium correspondence in Kohlberg & Mertens (1986), Govindan & Wilson (2003) attain a piece-wise differentiable path-following method to compute Nash equilibria. By constructing a convex-quadratic-penalty game, a differentiable path-following method is proposed in Chen & Dang (2019) for a refinement of the Nash equilibrium, which significantly outperforms a simplicial path-following method especially when the problem is large. As a result of the special structure of a two-person inspection game, Deutsch (2021) recently develops a polynomial-time method to compute all Nash equilibria solutions.

To the best of our knowledge, there is no method specifically designated for computing SSPEs in robust stochastic games. Kardeş et al. (2011) employ the LOQO software package to compute SSPEs in robust stochastic games. However, the package is for solving continuously differentiable constrained optimization problems with interior-point methods and a sequence of quadratic approximations and may fail to converge to an SSPE. Since the ambiguity sets of payoffs and transition probabilities in robust stochastic games induce a rather large number of variables, one can expect that simplicial path-following methods would take much more time than differentiable path-following methods especially when the problems are large. Inspired by this fact, this paper intends to develop a differentiable path-following method by capitalizing on special structures of robust stochastic games. Incorporating logarithmic-barrier terms into each player’s payoff function with an extra variable ranging between zero and one, we constitute a logarithmic-barrier robust stochastic game in which each player solves in each state against a given mixed strategy profile a convex optimization problem. An exploitation of the optimality conditions to the optimization problem together with a fixed point argument gives us a polynomial equilibrium system of the barrier game. The set of solutions to the equilibrium system contains a smooth path, which starts from an arbitrary given point and ends at an SSPE of the prime game as the extra variable descends from one to zero. As an alternative scheme, we also make up a convex-quadratic-penalty robust stochastic game and establish the existence of a smooth path to an SSPE. Numerical results show that the logarithmic-barrier differentiable path-following method significantly outperforms the convex-quadratic-penalty differentiable path-following method. To further evince its value, we apply the logarithmic-barrier differentiable path-following method to solve two robust stochastic games arising from supply chain configuration and market entry in area forwarding medical waste recycling.

The remaining of this paper is organized as follows. In Section 2, we introduce some notations in robust stochastic games and formulate the equilibrium system. In Section 3, we develop a logarithmic-barrier differentiable path-following method to find SSPEs in robust stochastic games. As an alternative scheme, a convex-quadratic-penalty differentiable path-following method is proposed in Section 4. Numerical performance and applications are reported in Section 5. We conclude this study in Section 6.