RPO-MAC: reciprocal Partially observable MAC protocol based on application-value-awareness in VANETs

Abstract

Numerous safety-related applications have rigid requirements for the threshold of latency limitation of message transmissions to enhance safety and traffic efficiency of vehicles on the road in vehicular ad-hoc networks (VANETs). Consequently, a medium access control (MAC) protocol for reliable and fast transmission of safety messages should focus on the maximum transmission delay. Herein, we refer to the concept of packet value and integrate it with the latency of different messages determined by the waiting interval of packets in a queue. Subsequently, we propose a reciprocal revenue function, i.e., the transmission probability of the participants, by considering both its own benefit and the other nodes’ revenue. Additionally, an inter-vehicle cross-layer cooperative game model based on the local optimal utility of participants is constructed. We then theoretically prove the existence of an equilibrium using the partially observable Markov decision process (POMDP), and provide a specific approach for obtaining the channel access probability of a vehicle by using deep reinforcement learning. Finally, the analysis and simulation results in saturated and non-saturated data traffic conditions are presented to evaluate the performance of Reciprocal Partially Observable MAC Protocol (RPO-MAC) proposed in this paper and compare it with the IEEE 802.11p standard protocol. These comparisons demonstrate the advantages of our proposed reciprocal revenue game method in the case of channel network congestion, especially in terms of delay. It is shown that the RPO-MAC protocol can provide a strong support to delay-sensitive safety related applications.

Appendices

Appendix A

Proof

When $t=1$ , it can be assumed that the initial channel usage state is idle, and obviously all primitive values $R_i^1$, $R_j^i$ of any two vehicles are independent of each other.

Suppose that when $t=k,k \in N^+$, the conclusion is true, i.e.,

$$\begin{aligned} {\mathbb {P}}^{\varvec{\psi }} \left( \varvec{r}^{\varvec{1:k}}|o^{1:k-1}\right) =\prod _{i=1}^n{\mathbb {P}}^{\phi _i}\left( r_i^{1:k}|o^{1:k-1}\right) . \end{aligned}$$

Then, when $t=k+1$,

$$\begin{aligned}&{\mathbb {P}}^{\varvec{\psi }} \left( \varvec{r^{1:k+1}},o^{1:k}\right) \\&\quad ={\mathbb {P}}\left( {o}^{1:k-1}\right) {\mathbb {P}}\left( \varvec{r}^{\varvec{1:k}}|{o}^{1:k-1}\right) {\mathbb {P}}\left( {o}^k|\varvec{r}^{\varvec{1:k}},{o}^{1:k-1}\right) *\\&\qquad {\mathbb {P}}^{\varvec{\psi }}\left( \varvec{r^{k+1}}|\varvec{r}^{\varvec{1:k}},{o}^{1:k}\right) \\&={\mathbb {P}}\left( {o}^{1:k-1}\right) {\mathbb {P}}\left( \varvec{r}^{\varvec{1:k}}|{o}^{1:k-1}\right) {\mathbb {P}}^{\varvec{\psi }}\left( {o}^{1:k}|\varvec{a^k}\right) \\&\qquad {\mathbb {P}}^{\varvec{\psi }}\left( \varvec{r^{k+1}}|\varvec{a^k},\varvec{r^k},{o}^{1:k}\right) \\&={\mathbb {P}}\left( {o}^{1:k-1}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) *{\mathbb {P}}\left( {o}^{1:k}|\varvec{r}^{\varvec{1:k}},{o}^{1:k-1}\right) \\&\qquad \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) \\&{\mathbb {P}}\left( r_i^{1:k+1},{o}^{1:k}\right) ={\mathbb {P}}\left( {o}^{1:k-1}\right) {\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k-1}\right) \\&\qquad {\mathbb {P}}\left( {o}^{1:k}|r_i^{1:k},{o}^{1:k-1}\right) {\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^{1:k},{o}^{1:k}\right) \\&{\mathbb {P}}\left( {o}^{1:k}\right) ={\mathbb {P}}\left( {o}^{1:k-1}\right) \ {\mathbb {P}}\left( {o}^{1:k}|{o}^{1:k-1}\right) \\ \end{aligned}$$

For any vehicle, i

$$\begin{aligned}&{\mathbb {P}}^\psi \left( r_i^{1:k+1}| o^{1:k}\right) \\&\quad =\dfrac{{\mathbb {P}}(r_i^{1:k+1},o^{1:k})}{{\mathbb {P}}(o^{1:k})}\\&\quad ={\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k-1}\right) {\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^{1:k},{o}^{1:k}\right) \\&\qquad \dfrac{{\mathbb {P}}\left( {o}^{1:k}|r_i^{1:k},{o}^{1:k-1}\right) }{{\mathbb {P}}\left( {o}^{1:k}|{o}^{1:k-1}\right) }\\&\quad ={\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k-1}\right) {\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^{1:k},{o}^{1:k}\right) \dfrac{{\mathbb {P}}\left( {o}^{1:k},r_i^{1:k}\right) }{{\mathbb {P}}\left( r_i^{1:k},{o}^{1:k-1}\right) }\\&\qquad \dfrac{{\mathbb {P}}({o}^{1:k-1})}{{\mathbb {P}}({o}^{1:k})}\\&\quad ={\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k-1}\right) {\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^{1:k},{o}^{1:k}\right) \dfrac{{\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k}\right) }{{\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k-1}\right) }\\&\quad ={\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^{1:k},{o}^{1:k}\right) {\mathbb {P}}\left( r_i^{1:k}|{o}^{1:k}\right) \\ \end{aligned}$$

So, when $t=k+1$,

$$\begin{aligned}&{\mathbb {P}}^{\varvec{\psi }} \left( \varvec{r^{1:k+1}}|o^{1:k}\right) \\&\quad =\dfrac{{\mathbb {P}}^{\varvec{\psi }} \left( \varvec{r^{1:k+1}},o^{1:k}\right) }{{\mathbb {P}}(o^{1:k})}\\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) *\\&\qquad \dfrac{{\mathbb {P}}^{\varvec{\psi }}\left( {o}^{1:k}|\varvec{r}^{\varvec{1:k}},{o}^{1:k-1}\right) }{{\mathbb {P}}\left( o^{1:k}|{o}^{1:k-1}\right) }\\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) *\\&\qquad \dfrac{{\mathbb {P}}^{\varvec{\psi }}\left( {o}^{1:k},\varvec{r}^{\varvec{1:k}}\right) }{{\mathbb {P}}^{\varvec{\psi }}\left( \varvec{r}^{\varvec{1:k}},{o}^{1:k-1}\right) }\frac{{\mathbb {P}}({o}^{1:k-1})}{{\mathbb {P}}({o}^{1:k})}\\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) *\\&\qquad \dfrac{{\mathbb {P}}^{\varvec{\psi }}\left( \varvec{r}^{\varvec{1:k}}|{o}^{1:k}\right) }{{\mathbb {P}}^{\varvec{\psi }}\left( \varvec{r}^{\varvec{1:k}}|{o}^{1:k-1}\right) }\\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) *\\&\qquad \dfrac{\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k}\right) }{\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k-1}\right) }\\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{k+1}|r_i^k,{o}^{1:k}\right) \prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k}|{o}^{1:k}\right) \\&\quad =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:k+1}|{o}^{1:k}\right) \\ \end{aligned}$$

Thus,

$$\begin{aligned} {\mathbb {P}}^{\varvec{\psi }} \left( \varvec{r^{1:t}}|o^{1:t-1}\right) =\prod _{i=1}^n{\mathbb {P}}^{\psi _i}\left( r_i^{1:t}|o^{1:t-1}\right) \end{aligned}$$

$\square$

Appendix B

Proof

$$\begin{aligned}&{\mathbb {P}}^{\varvec{\psi _{-i}}} \left( r_i^{t+1},o^{1:t}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right) \\&\quad ={\mathbb {P}}^{\varvec{\psi _{-i}}}\left( r_i^{t+1}|r_i^{1:t},o^{1:t},a_i^{1:t}\right) *{\mathbb {P}}^{\varvec{\psi _{-i}}} \left( o^{1:t}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right) \\&\quad ={\mathbb {P}}^{\varvec{\psi _{-i}}}\left( r_i^{t+1}|r_i^{t},o^{t},a_i^{t}\right) *{\mathbb {P}}^{\varvec{\psi _{-i}}} \left( \varvec{a_{-i}^t}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right) \\&\quad ={\mathbb {P}}^{\varvec{\psi _{-i}}}\left( r_i^{t+1}|r_i^{t},o^{t},a_i^{t}\right) *{\mathbb {P}}^{\varvec{\psi _{-i}}} \left( \varvec{a_{-i}^t}|o^{1:t-1}\right) \\&\quad ={\mathbb {P}}^{\varvec{\psi _{-i}}} \left( r_i^{t+1},o^{1:t}|r_i^{t},o^{1:t-1},a_i^{t}\right) \end{aligned}$$

where $a_i^t$ is a function of $r_i^t$. ${\mathbb {P}}^{\varvec{\psi _{-i}}}\left( \varvec{a_{-i}^t}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right)$ can be switched into ${\mathbb {P}}^{\varvec{\psi _{-i}}} \left( \varvec{a_{-i}^t}|o^{1:t-1}\right)$ because each action $a_i$ is conditionally independent given the historical observations $o^{1:t-1}$(Lemma 1).

For the second part,

$$\begin{aligned}&{\mathbb {E}}^{\varvec{\psi _{-i}}} \left\{ u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) |r_i^{1:t},o^{1:t-1},a_i^{1:t}\right\} \\&\quad =\sum _{\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}}u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) {\mathbb {P}}^{\varvec{\psi _{-i}}}\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right) \\&\quad =\sum _{\varvec{r_{-i}^t,a_{-i}^t}}u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) {\mathbb {P}}^{\varvec{\psi _{-i}}}\left( \varvec{r_{-i}^t,a_{-i}^t}|r_i^{1:t},o^{1:t-1},a_i^{1:t}\right) \\&\quad =\sum _{\varvec{r_{-i}^t,a_{-i}^t}}u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) {\mathbb {P}}^{\varvec{\psi _{-i}}}\left( \varvec{r_{-i}^t,a_{-i}^t}|o^{1:t-1}\right) \\&\quad ={\mathbb {E}}^{\varvec{\psi _{-i}}} \left\{ u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) |r_i^{t},o^{1:t-1},a_i^{t}\right\} \\&\quad =\hat{u_i}\left( r_i^{t},o^{1:t-1},a_i^{t}\right) \end{aligned}$$

$\square$

Appendix C

Proof

Fix $\varvec{\phi }$,

$$\begin{aligned}&b_i^{t+1}\left( r_i^{t+1}\right) ={\mathbb {P}} \left( r_i^{t+1}|o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}}}{\mathbb {P}} \left( r_i^{t+1},\varvec{r}^{\varvec{t}}|o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}}}{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}}|o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}^{\varvec{\psi }}\left( r_i^{t+1}|r_i^t,o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}}}{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}}|o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}\left( r_i^{t+1}|o^t,\gamma _i^t(r_i^t)\right) \end{aligned}$$

which

$$\begin{aligned}&{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}}|o^{1:t},\varvec{\gamma }^{\varvec{1:t}}\right) \\&\quad =\dfrac{{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}},o^t|o^{1:t-1}, \varvec{\gamma }^{\varvec{1:t}}\right) }{\sum _{\varvec{{\hat{r}}^t}}{\mathbb {P}}^{\varvec{\phi }} \left( \varvec{{\hat{r}}^t},o^t|o^{1:t-1},\varvec{\gamma }^{\varvec{1:t}}\right) }\\&\quad =\dfrac{{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}}|o^{1:t-1}, \varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}^{\varvec{\phi }} \left( o^t|o^{1:t-1},\varvec{\gamma }^{\varvec{1:t}},\varvec{r}^{\varvec{t}}\right) }{\sum _{\varvec{{\hat{r}}^t}}{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{{\hat{r}}^t}|o^{1:t-1}, \varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}^{\varvec{\phi }}\left( o^t|o^{1:t-1}, \varvec{\gamma }^{\varvec{1:t}},\varvec{{\hat{r}}^t}\right) }\\&\quad =\dfrac{{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{r}^{\varvec{t}}|o^{1:t-1},\varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}^{\varvec{\phi }}\left( o^t|\varvec{\gamma ^{t}(r^t)}\right) }{\sum _{\varvec{{\hat{r}}^t}}{\mathbb {P}}^{\varvec{\phi }}\left( \varvec{{\hat{r}}^t}|o^{1:t-1},\varvec{\gamma }^{\varvec{1:t}}\right) {\mathbb {P}}^{\varvec{\phi }}\left( o^t|\varvec{\gamma ^{t}({\hat{r}}^t)}\right) } \end{aligned}$$

Since packet values of all vehicles $R_i^{1:t}$ are conditionally independent with the observations $o^{1:t-1}$ (Lemma 1), we can get:

$$\begin{aligned} {\mathbb {P}}^{\varvec{\phi }}(\varvec{r}^{\varvec{t}}|{o}^{1:t},\varvec{\gamma }^{\varvec{1:t}})=\dfrac{\prod _{i=1}^{N}{b_i^t(r_i^t)}{\mathbb {P}}^{\varvec{\phi }}\left( {o}^t|\varvec{\gamma }^{\varvec{t}}(\varvec{r}^{\varvec{t}})\right) }{\sum _{\varvec{{\hat{r}}^t}}{\prod _{i=1}^{N}{b_i^t({{\hat{r}}}_i^t)}{\mathbb {P}}^{\varvec{\phi }}({o}^t|\varvec{\gamma }^{\varvec{t}}({\varvec{{\hat{r}}^t}}))}} \end{aligned}$$

So, $b_i^{t+1}$ can be expressed as follows:

$$\begin{aligned}&b_i^{t+1}\left( r_i^{t+1}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}}}{\mathbb {P}}\left( r_i^{t+1}|o^t,\gamma _i^t(r_i^t)\right) \dfrac{\prod _{i=1}^{N}{b_i^t(r_i^t)}{\mathbb {P}}^{\varvec{\phi }}\left( {o}^t|\varvec{\gamma }^{\varvec{t}}(\varvec{r}^{\varvec{t}})\right) }{\sum _{\varvec{{\hat{r}}^t}}{\prod _{i=1}^{N}{b_i^t({{\hat{r}}}_i^t)}{\mathbb {P}}^{\varvec{\phi }}({o}^t|\varvec{\gamma }^{\varvec{t}}({\varvec{{\hat{r}}^t}}))}} \end{aligned}$$

We can further simplify this function because the delay of vehicle i in time $t + 1$ is meaningful only when it chooses to wait at time t.Thus,

$$\begin{aligned}&b_i^{t+1}\left( r_i^{t+1}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}}}\varvec{1}_{\left\{ W\right\} }({\gamma _i^t(r_i^t)}){\mathbb {P}}\left( r_i^{t+1}|A_i = W,r_i^t,{o}^t\right) \\&\qquad \dfrac{\prod _{i=1}^{N}{b_i^t(r_i^t)}{\mathbb {P}}^{\varvec{\phi }}\left( {o}^t|A_i = W,\varvec{\gamma _{-i}^t}(\varvec{r_{-i}^t})\right) }{\sum _{\varvec{{\hat{r}}^t}}{\prod _{i=1}^{N}{b_i^t({{\hat{r}}}_i^t)}{\mathbb {P}}^{\varvec{\phi }}({o}^t|A_i = W,\varvec{\gamma _{-i}^t}({\varvec{{\hat{r}}^t_{-i}}}))}}\\&\quad =\dfrac{\sum _{r_i^t}\varvec{1}_{\left\{ W\right\} }({\gamma _i^t(r_i^t)}){\mathbb {P}}\left( r_i^{t+1}|A_i=W,r_i^t,{o}^t\right) b_i^t(r_i^t)\sum _{\varvec{r_{-i}^t}}\prod _{j=1,j\not =i}^{N}b_j^t(r_j^t){\mathbb {P}}\left( {o}^t|\varvec{\gamma _{-i}^t}(\varvec{r_{-i}^t})\right) }{\sum _{{\hat{r}}_i^t}b_i^t({\hat{r}}_i^t)\sum _{\varvec{{\hat{r}}_{-i}^t}}\prod _{j=1,j\not =i}^{N}{b_j^t({{\hat{r}}}_j^t)}{\mathbb {P}}({o}^t|\varvec{\gamma _{-i}^t}({\varvec{{\hat{r}}^t_{-i}}}))}\\&\quad =\dfrac{\sum _{r_i^t}\varvec{1}_{\left\{ W\right\} }({\gamma _i^t(r_i^t)}){\mathbb {P}}\left( r_i^{t+1}|A_i=W,r_i^t,{o}^t\right) b_i^t(r_i^t)}{\sum _{{\hat{r}}_i^t}b_i^t({\hat{r}}_i^t)}\\&\quad =H_i\left( b_i^t,\gamma _i^t,o^t\right) \end{aligned}$$

where $\varvec{1}(\cdot )$ is the indicator function and H is independent of policy $\psi$. $\square$

Appendix D

Proof

$$\begin{aligned}&{\mathbb {P}}\left( \varvec{b^{t+1}}|\varvec{b^{1:t}},\varvec{\gamma }^{\varvec{1:t}}\right) \\&\quad =\sum _{{o}^t}{\mathbb {P}}\left( \varvec{b}^{\varvec{t}+{\mathbf {1}}},{o}^t|\varvec{b}^{{\mathbf {1}}:\varvec{t}},\varvec{\gamma }^{{\mathbf {1}}:\varvec{t}}\right) \\&\quad =\sum _{{o}^t}\varvec{1}_{\left\{ H\left( \varvec{b}^{\varvec{t}},\varvec{\gamma }^{\varvec{t}},{o}^t\right) \right\} }(\varvec{b^{t+1}}){\mathbb {P}}\left( {o}^t|\varvec{b}^{{\mathbf {1}}:\varvec{t}},\varvec{\gamma }^{{\mathbf {1}}:\varvec{t}}\right) \\&\quad =\sum _{{o}^t,\varvec{r}^{\varvec{t}}}\varvec{1}_{\left\{ H\left( \varvec{b}^{\varvec{t}},\varvec{\gamma }^{\varvec{t}},{o}^t\right) \right\} }(\varvec{b^{t+1}}){\mathbb {P}}\left( {o}^t|\varvec{\gamma ^t(r^t)}\right) {\mathbb {P}}\left( \varvec{r}^{\varvec{t}}|\varvec{b}^{{\mathbf {1}}:\varvec{t}},\varvec{\gamma }^{{\mathbf {1}}:\varvec{t}}\right) \\&\quad =\sum _{{o}^t,\varvec{r}^{\varvec{t}}}\varvec{1}_{\left\{ H\left( \varvec{b}^{\varvec{t}},\varvec{\gamma }^{\varvec{t}},{o}^t\right) \right\} }(\varvec{b^{t+1}}){\mathbb {P}}\left( {o}^t|\varvec{\gamma ^t(r^t)}\right) \prod _{i=1}^{N}{b_i(r_i^t)}\\&\quad ={\mathbb {P}}\left( \varvec{b^{t+1}}|\varvec{b^{t}},\varvec{\gamma }^{\varvec{t}}\right) \\&{\mathbb {E}}\left\{ u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) |\varvec{b^{1:t}},\varvec{\gamma }^{\varvec{1:t}}\right\} \\&\quad =\sum _{\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}}u_i(\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}){\mathbb {P}}\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}|\varvec{b^{1:t},\gamma ^{1:t}}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}}u_i(\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}})\varvec{1}_{\left\{ \varvec{\gamma ^t(r^t)}\right\} }(\varvec{a^t}){\mathbb {P}}\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}|\varvec{b^{1:t},\gamma ^{1:t}}\right) \\&\quad =\sum _{\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}}u_i(\varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}})\varvec{1}_{\left\{ \varvec{\gamma ^t(r^t)}\right\} }(\varvec{a^t})\prod _{i=1}^{N}{b_i(r_i^t)}\\&\quad ={\mathbb {E}} \left\{ u_i\left( \varvec{r}^{\varvec{t}},\varvec{a}^{\varvec{t}}\right) |\varvec{b^{t}},\varvec{\gamma }^{\varvec{t}}\right\} \\&\quad =\hat{u_i}\left( \varvec{b^{t}},\varvec{\gamma }^{\varvec{t}}\right) \\ \end{aligned}$$

$\square$