The most prominent challenge to the wireless community is to meet the demand for radio resources. Cognitive Radio (CR) is envisioned as a potential solution that utilizes its cognition ability intended to enhance the proper utilization of available radio resources and improves energy efficiency. However, due to the co-existence of Primary Base Stations (PU-BSs) and Cognitive Base Stations (CR-BSs) in CR networks, the problem of aggregated interference occurs which poses a critical challenge for resource allocation in CR networks. Moreover, in practical scenarios, it is difficult to form the correct network model due to complex network dynamics beforehand. Therefore, this work presents Multi-Agent Model-Free Reinforcement Learning schemes namely Q-Learning (Q-L) and State-Action-Reward- (next) State- (next) Action (SARSA) for resource allocation which mitigates interference and eliminate the need of network model. The proposed schemes are implemented in a decentralized cooperative manner with CRs act as multi-agent, forms a stochastic dynamic team to obtain optimal energy-efficient resource allocation strategy. Numerical results reveal that: 1) proposed cooperative scheme 1 (Cooperative Q-L scheme) expedites the convergence; 2) proposed cooperative scheme 2 (Cooperative SARSA scheme) achieves significant improvement in network capacity. Both the proposed cooperative schemes demonstrate its effectiveness by providing significant improvement in energy efficiency and maintain users’ QoS.

中文翻译：

---

协作多智能体无模型强化学习方案下认知无线电网络的节能资源分配


无线社区面临的最突出的挑战是满足无线资源的需求。认知无线电（CR）被设想为一种潜在的解决方案，利用其认知能力来增强可用无线电资源的正确利用并提高能源效率。然而，由于CR网络中主基站（PU-BS）和认知基站（CR-BS）共存，出现聚集干扰问题，这对CR网络的资源分配提出了严峻的挑战。而且，在实际场景中，由于事先复杂的网络动态，很难形成正确的网络模型。因此，这项工作提出了多智能体无模型强化学习方案，即 Q-Learning (QL) 和 State-Action-Reward-(next) State-(next) Action (SARSA)，用于资源分配，从而减轻干扰并消除需求的网络模型。所提出的方案以分散协作的方式实现，CR作为多智能体，形成随机动态团队以获得最优的节能资源分配策略。数值结果表明：1）提出的合作方案1（Cooperative QL方案）加快了收敛速度； 2）提出的合作方案2（Cooperative SARSA方案）实现了网络容量的显着提升。所提出的两种协作方案都通过显着提高能源效率并维持用户的服务质量来证明其有效性。