The explosive growth of Internet-of-Things (IoT) applications such as smart cities and Industry 4.0 have led to drastic increase in demand for wireless bandwidth, hence motivating the rapid development of new techniques for enhancing spectrum utilization needed by new generation wireless communication technologies. Among others, dynamic spectrum access (DSA) is one of the most widely accepted approaches. In this paper, as an enhancement of existing works, we take into consideration of inter-node collaborations in a dynamic spectrum environment. Typically, in such distributed circumstances, intelligent dynamic spectrum access almost invariably relies on self-learning to achieve dynamic spectrum access improvement. Whereas, this paper proposes a DSA scheme based on deep reinforcement learning to enhance spectrum and access efficiency. Unlike traditional Q-learning-based DSA, we introduce the following to enhance the spectrum efficiency in dynamic IoT spectrum environments. First, deep double Q-learning is adopted to perform local self-spectrum-learning for IoT terminals in order to achieve better dynamic access accuracy. Second, to accelerate learning convergence, federated learning (FL) in edge nodes is used to improve the self-learning. Third, multiple secondary users, who do not interfere with each other and have similar operation condition, are clustered for federated learning to enhance the efficiency of deep reinforcement learning. Comparing with the traditional distributed DSA with deep learning, the proposed scheme has faster access convergence speed due to the characteristic of global optimization for federated learning. Based on this, a framework of federated deep reinforcement learning (FDRL) for DSA is proposed. Furthermore, this scheme preserves privacy of IoT users in that FDRL only requires model parameters to be uploaded to edge servers. Simulations are performed to show the effectiveness of theproposed FDRL-based DSA framework.

中文翻译：

---

基于联邦深度强化学习的物联网动态频谱接入


智慧城市和工业4.0等物联网（IoT）应用的爆炸式增长导致对无线带宽的需求急剧增加，从而推动了新技术的快速发展，以提高新一代无线通信技术所需的频谱利用率。其中，动态频谱接入（DSA）是最广泛接受的方法之一。在本文中，作为现有工作的增强，我们考虑了动态频谱环境中的节点间协作。通常，在这种分布式环境下，智能动态频谱接入几乎总是依赖自学习来实现动态频谱接入改进。鉴于此，本文提出了一种基于深度强化学习的DSA方案，以提高频谱和接入效率。与传统的基于 Q-learning 的 DSA 不同，我们引入以下内容来提高动态物联网频谱环境中的频谱效率。首先，采用深度双Q学习对物联网终端进行本地自频谱学习，以达到更好的动态访问精度。其次，为了加速学习收敛，边缘节点使用联邦学习（FL）来提高自学习能力。第三，将多个互不干扰、操作情况相似的二级用户聚集起来进行联邦学习，以提高深度强化学习的效率。与传统的深度学习分布式DSA相比，由于联邦学习全局优化的特性，所提出的方案具有更快的访问收敛速度。在此基础上，提出了用于DSA的联邦深度强化学习（FDRL）框架。 此外，该方案保护了物联网用户的隐私，因为 FDRL 仅需要将模型参数上传到边缘服务器。进行仿真以显示所提出的基于 FDRL 的 DSA 框架的有效性。