A Markovian Stackelberg game approach for computing an optimal dynamic mechanism

Abstract

This paper presents a dynamic Bayesian–Stackelberg incentive-compatible mechanism, in which multiple agents observe private information and learn their behavior through a sequence of interactions in a repeated game, for a class of controllable homogeneous Markov games. We assume that the leaders can ex ante commit to their disclosure strategy and mechanism, and affect followers’ actions. Along the paper, leaders possess and benefit from some commitment leadership, which describes the distinctive nature of a Stackelberg game. In this dynamics, leaders and followers together are in a Stackelberg game where actions are taken in a sequential way in the two layers of the hierarchy, but independently leaders and followers are involved non-cooperativelyin two (Nash) games where actions are taken simultaneously. This game considers an ex-ante incentive-compatible mechanism, which in equilibrium maximizes the reward while the agents are learning their actions over a countable number of periods. The formulation of the problem considers a Bayesian–Stackelberg equilibrium in the context of Reinforcement Learning. We propose an algorithm supported by the extraproximal method and show that it converges. The Tikhonov’s regularization technique is employed for ensuring the existence and uniqueness of the Bayesian–Stackelberg equilibrium. We show and guarantee the convergence of the method to a single incentive-compatible mechanism. We derive the analytical expressions for computing the mechanism in a Stackelberg game, which is one of the main results of this work. We demonstrate the efficiency of the method by an experiment drawn from an electric power problem represented by an oligopolistic market structure dominated by a small number of large sellers (oligopolists).

A Proof of Theorem 2

.

Following (Antipin 2005; Trejo et al. 2015a), let us consider $\eta =\gamma , \text { } z={\tilde{w}},\text { }x= {\tilde{v}}_{n}$ and $z^{*} =\hat{v}_{n}$

$$\begin{aligned} \begin{array}{c} r(z)={\mathcal {L}}_{\delta }({\tilde{w}},{\tilde{v}}_{n}), \text { } r(z^{*})={\mathcal {L}}_{\delta }(\hat{v}_{n} ,{\tilde{v}}_{n}) \end{array} \end{aligned}$$

we obtain

$$\begin{aligned} \begin{array}{c} \dfrac{1}{2}\Vert \hat{v}_{n} -{\widetilde{v}}_{n} \Vert ^{2} +\gamma {\mathcal {L}}_{\delta } (\hat{v}_{n},{\tilde{v}}_{n} )\le \dfrac{1}{2} \Vert {\tilde{w}}-{\tilde{v}}_{n}\Vert ^{2} + \gamma {\mathcal {L}}_{\delta }({\tilde{w}},{\tilde{v}}_{n} )-\dfrac{1}{2} \Vert {\tilde{w}}-\hat{v}_{n}\Vert ^{2} \end{array} \end{aligned}$$

(34)

Now, considering $\eta =\gamma , \small z={\tilde{w}},\text { }x={\tilde{v}}_{n}$ and $z^{*}={\tilde{v}}_{n+1}$

$$\begin{aligned} \begin{array}{c} r(z)={\mathcal {L}}_{\delta }({\tilde{w}},\hat{v}_{n} ),\text { } r(z^{*})={\mathcal {L}}_{\delta } ({\tilde{v}}_{n+1},\hat{v}_{n}) \end{array} \end{aligned}$$

we obtain

$$\begin{aligned} \begin{array}{c} \dfrac{1}{2}\Vert {\tilde{v}}_{n+1} -{\tilde{v}}_{n}\Vert ^{2}+\gamma {\mathcal {L}}_{\delta } ({\tilde{v}}_{n+1},\hat{v}_{n} )\le \dfrac{1}{2} \Vert {\tilde{w}}-{\tilde{v}}_{n} \Vert ^{2} +\gamma {\mathcal {L}}_{\delta } ({\tilde{w}},\hat{v}_{n} )-\dfrac{1}{2} \Vert {\tilde{w}}-{\tilde{v}}_{n+1} \Vert ^{2} \end{array} \quad \end{aligned}$$

(35)

Choosing ${{\tilde{w}}}={\tilde{v}}_{n+1}$ in (34) and ${{\tilde{w}} }= \hat{v}_{n}$ in (35) we obtain

$$\begin{aligned}&\begin{array}{c} \dfrac{1}{2}\Vert \hat{v}_{n}-{\tilde{v}}_{n} \Vert ^{2} +\gamma {\mathcal {L}}_{\delta } (\hat{v}_{n},{\tilde{v}}_{n})\le \dfrac{1}{2} \Vert {\tilde{v}}_{n+1}-{\tilde{v}}_{n} \Vert ^{2}+ \gamma {\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},{\tilde{v}}_{n} )-\dfrac{1}{2} \Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2} \end{array} \nonumber \\\end{aligned}$$

(36)

$$\begin{aligned}&\begin{array}{c} \dfrac{1}{2}\Vert {\tilde{v}}_{n+1} -{\tilde{v}}_{n}\Vert ^{2}+ \gamma {\mathcal {L}}_{\delta } ({\tilde{v}}_{n+1},\hat{v}_{n}) \le \dfrac{1}{2} \Vert \hat{v}_{n}-{\tilde{v}}_{n} \Vert ^{2} +\gamma {\mathcal {L}}_{\delta } (\hat{v}_{n},\hat{v}_{n}) -\dfrac{1}{2} \Vert \hat{v}_{n} -{\tilde{v}}_{n+1} \Vert ^{2} \end{array} \nonumber \\ \end{aligned}$$

(37)

Adding (36) with (37) and considering ${\tilde{w}}+h={\tilde{v}}_{n+1}\text { }{\tilde{w}}=\hat{v}_{n},\text { }{\tilde{v}}+q={\tilde{v}}_{n},\text { }{\tilde{v}}=\hat{ v}_{n}, h={\tilde{v}}_{n+1}-\hat{v}_{n}$ and $q={\tilde{v}}_{n} -\hat{v}_{n}$ then, we have

$$\begin{aligned} \begin{array}{l} \Vert {\tilde{v}}_{n+1}-\hat{v}_{n}\Vert ^{2}\le \gamma [{\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},{\tilde{v}}_{n} )-{\mathcal {L}}_{\delta } (\hat{v}_{n},{\tilde{v}}_{n} )] \\ \quad -\gamma [{\mathcal {L}}_{\delta }({\tilde{v}}_{n+1} , \hat{v}_{n})-{\mathcal {L}}_{\delta } (\hat{v}_{n} ,\hat{v}_{n})]\le \gamma L\Vert {\tilde{v}} _{n+1}-\widehat{v}_{n} \Vert \Vert {\tilde{v}}_{n}-\hat{v}_{n} \Vert \end{array} \end{aligned}$$

which implies

$$\begin{aligned} \Vert {\tilde{v}}_{n+1}- \hat{v}_{n}\Vert \le \gamma L\Vert {\tilde{v}}_{n}- \hat{v}_{n}\Vert \end{aligned}$$

(38)

Now, considering ${\tilde{w}}={\tilde{v}}_{n+1}$ in (34) and ${\tilde{w}}= {\tilde{v}}_{\delta }^{*}$ in (35) we have

$$\begin{aligned} \begin{array}{c} \dfrac{1}{2}\Vert \hat{v}_{n}-{\tilde{v}}_{n} \Vert ^{2}+\gamma {\mathcal {L}}_{\delta }(\hat{v}_{n},{\tilde{v}}_{n} )\le \frac{1}{2} \Vert {\tilde{v}}_{n+1} -{\tilde{v}} _{n} \Vert ^{2}+ \gamma {\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},{\tilde{v}}_{n})-\dfrac{1}{2}\Vert {\tilde{v}}_{n+1}-\hat{v}_{n}\Vert ^{2} \end{array} \\ \begin{array}{c} \dfrac{1}{2}\Vert {\tilde{v}}_{n+1}-{\tilde{v}}_{n}\Vert ^{2}+\gamma {\mathcal {L}}_{\delta }({\tilde{v}}_{n+1} ,\hat{v}_{n} )\le \frac{1}{2} \Vert {\tilde{v}}_{\delta }^{*} -{\tilde{v}}_{n} \Vert ^{2} + \gamma {\mathcal {L}}_{\delta }({\tilde{v}}_{\delta }^{*},\hat{v}_{n})-\frac{1}{2}\Vert {\tilde{v}}_{\delta }^{*} -{\tilde{v}}_{n+1} \Vert ^{2} \end{array} \end{aligned}$$

Adding the previous inequalities and multiplying by two we obtain

$$\begin{aligned} \begin{array}{l} \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}+\Vert {\tilde{v}}_{n+1} -\hat{v} _{n}\Vert ^{2}+\Vert \hat{v}_{n}-{\tilde{v}}_{n}\Vert ^{2}-2\gamma {\mathcal {L}}_{\delta }({\tilde{v}}_{\delta }^{*},\hat{v}_{n})\\ \quad +2\gamma [{\mathcal {L}} _{\delta }( {\tilde{v}}_{n+1},\hat{v}_{n} ) + {\mathcal {L}}_{\delta }(\hat{v}_{n},{\tilde{v}}_{n})-{\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},{\tilde{v}}_{n} )] \text { }\le \text { }\Vert {\tilde{v}}_{\delta }^{*} -{\tilde{v}}_{n}\Vert ^{2} \end{array} \end{aligned}$$

Adding and subtracting the term ${\mathcal {L}}_{\delta }(\hat{v}_{n}, \hat{v}_{n})$ we get

$$\begin{aligned} \begin{array}{l} \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}+\Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2}+\Vert \hat{v}_{n} -{\tilde{v}}_{n} \Vert ^{2}+ 2\gamma \left[ {\mathcal {L}}_{\delta }(\hat{v}_{n},\hat{v}_{n})-{\mathcal {L}}_{\delta }({\tilde{v}}_{\delta }^{*},\hat{v}_{n})\right] \\ \quad + 2\gamma \left[ {\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},\hat{v}_{n}) -{\mathcal {L}}_{\delta }(\hat{v}_{n},\hat{v}_{n})+{\mathcal {L}}_{\delta } (\hat{v}_{n}, {\tilde{v}}_{n} )-{\mathcal {L}}_{\delta }({\tilde{v}}_{n+1},{\tilde{v}}_{n} )\right] \le \text { }\Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{array} \end{aligned}$$

Considering ${\tilde{w}}+h={\tilde{v}}_{n+1}$, ${\tilde{w}}=\hat{v}_{n}$, ${\tilde{v}}+k= {\tilde{v}}_{n}$ and ${\tilde{v}}=\hat{v}_{n}$ we have $h={\tilde{v}}_{n+1}- \hat{v}_{n}$ and $k={\tilde{v}}_{n}-\hat{v}_{n}$, then the resulting equation is as follows

$$\begin{aligned} \begin{array}{l} \Vert {\tilde{v}}_{\delta }^{*} -{\tilde{v}}_{n+1}\Vert ^{2}+\Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2}+\Vert \hat{v}_{n}-{\tilde{v}}_{n} \Vert ^{2}+ 2\gamma \left[ {\mathcal {L}}_{\delta } (\hat{v}_{n},\hat{v}_{n})-{\mathcal {L}}_{\delta }({\tilde{v}}_{\delta }^{*},\hat{v}_{n})\right] \\ \quad - 2\gamma L\Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert \Vert {\tilde{v}}_{n} -\hat{v}_{n} \Vert \le \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{array} \end{aligned}$$

Using Eq. (38) in the last term on the left-hand side and given the strict convexity of ${\mathcal {L}}_{\delta }$ where

$$\begin{aligned} {\mathcal {L}}_{\delta }(\hat{v}_{n},\hat{v}_{n})-{\mathcal {L}}_{\delta }( {\tilde{v}}_{\delta }^{*},\hat{v}_{n})\ge \delta \Vert \hat{v}_{n} - {\tilde{v}}_{\delta }^{*}\Vert ^{2} \end{aligned}$$

we obtain

$$\begin{aligned}&\Vert {\tilde{v}}_{\delta }^{*} -{\tilde{v}}_{n+1}\Vert ^{2} +\Vert {\tilde{v}}_{n+1}-\hat{v}_{n}\Vert ^{2}+2\gamma \delta \Vert \hat{v}_{n}-{\tilde{v}}_{\delta }^{*}\Vert ^{2}\\&\qquad +\left( 1-2\gamma ^{2}L^{2}\right) \Vert {\tilde{v}}_{n} -\hat{v}_{n}\Vert ^{2} \le \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{aligned}$$

Applying the identity $2\langle a-c,c-b\rangle = \Vert a-b\Vert ^{2}- \Vert a-c\Vert ^{2} -\Vert c-b\Vert ^{2}$ with $a=\hat{v}_{n}$, $b={\tilde{v}}_{\delta }^{*}$ and $c={\tilde{v}}_{n},$ to the left-hand side of the last inequality we have

$$\begin{aligned} \begin{array}{l} \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}+\Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2}+\left( 1-2\gamma ^{2}L^{2}\right) \Vert {\tilde{v}}_{n}-\hat{v}_{n} \Vert ^{2} + 2\gamma \delta [2\langle \hat{v}_{n}-{\tilde{v}}_{n},{\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*}\rangle ]\\ \quad +\Vert {\tilde{v}}_{n} -\hat{v}_{n}\Vert ^{2}+ \Vert {\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*} \Vert ^{2}= \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2} +\Vert {\tilde{v}}_{n+1} -\hat{v}_{n} \Vert ^{2} \\ \qquad + (1+2\gamma \delta -2\gamma ^{2}L^{2} )\Vert {\tilde{v}}_{n}-\hat{v}_{n}\Vert ^{2}\\ \quad +4\gamma \delta \langle \hat{v}_{n}-{\tilde{v}}_{n},{\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*} \rangle +2\gamma \delta \Vert {\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*}\Vert ^{2}\le \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{array} \end{aligned}$$

Let $p=1+2 \gamma \delta -2 \gamma ^{2}L^{2}$ and considering the square form of the third and fourth terms yields

$$\begin{aligned}&\Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}+ \Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2}+ p\Vert {\tilde{v}}_{n}-\hat{v}_{n}\Vert ^{2}+ 4\gamma \delta \langle \hat{v}_{n}-{\tilde{v}}_{n},{\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*}\rangle \\&\quad +\dfrac{(2\gamma \delta )^{2}}{p}\Vert {\tilde{v}}_{n} -{\tilde{v}}_{\delta }^{*}\Vert ^{2}-\dfrac{(2\gamma \delta )^{2}}{p}\Vert {\tilde{v}}_{n} -{\tilde{v}}_{\delta }^{*}\Vert ^{2}+ 2\gamma \delta \Vert {\tilde{v}}_{n} -{\tilde{v}}_{\delta }^{*}\Vert ^{2}\le \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{aligned}$$

and

$$\begin{aligned}&\Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}+ \Vert {\tilde{v}}_{n+1} -\hat{v}_{n}\Vert ^{2} + \left\| \sqrt{ p}({\tilde{v}}_{n}-\hat{v}_{n})+ \frac{ 2\gamma \delta }{\sqrt{p}}({\tilde{v}}_{n}-{\tilde{v}}_{\delta }^{*})\right\| ^{2}\\&\quad \le \left( 1-2\gamma \delta +\frac{(2\gamma \delta )^{2}}{p}\right) \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2} \end{aligned}$$

considering $k=1-2\gamma \delta +\dfrac{(2\gamma \delta )^{2}}{p}\in \left( 0,1\right)$ we have that

$$\begin{aligned} \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n+1}\Vert ^{2}\le k \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{n}\Vert ^{2}\le k^{n+1} \Vert {\tilde{v}}_{\delta }^{*}-{\tilde{v}}_{0}\Vert ^{2}\underset{n\rightarrow \infty }{\rightarrow }0. \end{aligned}$$

q.e.d..