In this paper, we study the channel access problem of vehicles in a cognitive radio vehicular network, where each vehicle opportunistically accesses the channel resources of the primary network in order to successfully receive the necessary data packets within a time deadline. Given the access priority constraint and the limited bandwidth of the primary network, a smart channel connection scheme is indispensable to ensure a decent quality of service (QoS) at the vehicles’ side. Due to the competitive nature of vehicles, the vehicle access control is formulated as a multi-agent access problem that comes with an intrinsic challenge, i.e. the partial observation of the information about the environment dynamics. On top of that, considering the temporal usage profile of the primary network, the environment dynamics are also time-dependant, and hence making the aforementioned access control a non-Markovian problem. Consequently, the estimation of the system states, which are used for the decision making process of a vehicle, is very challenging. To deal with the issues arising from such non-Markovian problem, we propose a vehicle connection algorithm based on a deep recurrent Q-learning network. With the aid of a recurrent Long Short Term Memory (LSTM) layer integrated into a deep Q-network, the time-correlated system states can be properly estimated, thereby improving the vehicle channel access policy. Besides, we introduce novel reward quantities that help improving the network performance and the capability to flexibly adapt to unexplored scenarios. A new structure of the cumulative reward function is also presented to balance the performance trade off between the cooperative and competitive objectives. Simulation results are provided to verify the advantage and the stability of our proposed algorithm over the benchmark schemes.

中文翻译：

---

具有多代理设置的认知车辆网络中基于 LSTM 的车辆通道访问方案


在本文中，我们研究了认知无线电车载网络中车辆的信道访问问题，其中每个车辆机会性地访问主网络的信道资源，以便在期限内成功接收必要的数据包。考虑到接入优先级约束和主网络带宽有限，为了确保车辆侧良好的服务质量（QoS），智能通道连接方案是必不可少的。由于车辆的竞争性质，车辆访问控制被表述为一个多智能体访问问题，它带来了固有的挑战，即对环境动态信息的部分观察。最重要的是，考虑到主网络的时间使用情况，环境动态也是时间相关的，因此使上述访问控制成为非马尔可夫问题。因此，用于车辆决策过程的系统状态的估计非常具有挑战性。为了解决此类非马尔可夫问题引起的问题，我们提出了一种基于深度循环 Q 学习网络的车辆连接算法。借助集成到深度 Q 网络中的循环长短期记忆 (LSTM) 层，可以正确估计时间相关的系统状态，从而改进车辆通道访问策略。此外，我们引入了新颖的奖励数量，有助于提高网络性能和灵活适应未探索场景的能力。还提出了累积奖励函数的新结构，以平衡合作目标和竞争目标之间的绩效权衡。 提供仿真结果来验证我们提出的算法相对于基准方案的优势和稳定性。