In order to meet the growing demands for multimedia service access and release the pressure of the core network, edge caching and device-to-device (D2D) communication have been regarded as two promising techniques in next generation mobile networks and beyond. However, most existing related studies lack consideration of effective cooperation and adaptability to the dynamic network environments. In this article, based on the flexible trilateral cooperation among user equipment, edge base stations and a cloud server, we propose a D2D-assisted heterogeneous collaborative edge caching framework by jointly optimizing the node selection and cache replacement in mobile networks. We formulate the joint optimization problem as a Markov decision process, and use a deep Q-learning network to solve the long-term mixed integer linear programming problem. We further design an attention-weighted federated deep reinforcement learning (AWFDRL) model that uses federated learning to improve the training efficiency of the Q-learning network by considering the limited computing and storage capacity, and incorporates an attention mechanism to optimize the aggregation weights to avoid the imbalance of local model quality. We prove the convergence of the corresponding algorithm, and present simulation results to show the effectiveness of the proposed AWFDRL framework in reducing average delay of content access, improving hit rate and offloading traffic.

中文翻译：

---

用于设备到设备辅助异构协作边缘缓存的注意力加权联合深度强化学习

为了满足日益增长的多媒体业务接入需求，缓解核心网的压力，边缘缓存和设备到设备（D2D）通信被视为下一代移动网络及以后的两种有前途的技术。然而，现有的相关研究大多缺乏对动态网络环境的有效协作和适应性的考虑。在本文中，基于用户设备、边缘基站和云服务器之间灵活的三边协作，我们通过联合优化移动网络中的节点选择和缓存替换，提出了一种 D2D 辅助的异构协同边缘缓存框架。我们将联合优化问题表述为马尔可夫决策过程，并使用深度 Q 学习网络解决长期混合整数线性规划问题。我们进一步设计了一个注意力加权联邦深度强化学习（AWFDRL）模型，该模型通过考虑有限的计算和存储容量，使用联邦学习来提高 Q-learning 网络的训练效率，并结合一个注意力机制来优化聚合权重以避免局部模型质量的不平衡。我们证明了相应算法的收敛性，并通过仿真结果展示了所提出的 AWFDRL 框架在降低内容访问平均延迟、提高命中率和卸载流量方面的有效性。并结合了一种注意力机制来优化聚合权重，以避免局部模型质量的不平衡。我们证明了相应算法的收敛性，并通过仿真结果展示了所提出的 AWFDRL 框架在减少内容访问的平均延迟、提高命中率和卸载流量方面的有效性。并结合了一种注意力机制来优化聚合权重，以避免局部模型质量的不平衡。我们证明了相应算法的收敛性，并通过仿真结果展示了所提出的 AWFDRL 框架在减少内容访问的平均延迟、提高命中率和卸载流量方面的有效性。