Network function virtualization (NFV) is critical to the scalability and flexibility of various network services in the form of service function chains (SFCs), which refer to a set of Virtual Network Functions (VNFs) chained in a specific order. However, the NFV performance is hard to fulfill the ever-increasing requirements of network services mainly due to the static orchestrations of SFCs. To tackle this issue, a novel Scalable SFC Orchestration (SSCO) scheme is proposed in this paper for NFV-enabled networks via federated reinforcement learning. SSCO has three remarkable characteristics distinguishing from the previous work: (1) A federated-learning-based framework is designed to train a global learning model, with time-variant local model explorations, for scalable SFC orchestration, while avoiding data sharing among stakeholders; (2) SSCO allows for parameter update among local clients and the cloud server just at the first and last epochs of each episode to ensure that distributed clients can make model optimization at a low communication cost; (3) SSCO introduces an efficient deep reinforcement learning (DRL) approach, with the local learning knowledge of available resources and instantiation cost, to map VNFs into networks flexibly. Furthermore, a loss-weight-based mechanism is proposed to generate and exploit reference samples in replay buffers for future training, avoiding the strong relevance of samples. Simulation results obtained from different working scenarios demonstrate that SSCO can significantly reduce placement errors and improve resource utilization ratio to place time-variant VNFs compared with the state-of-the-art mechanisms. Furthermore, the results show that the proposed approach can achieve desirable scalability.

中文翻译：

---

支持 NFV 的网络中服务功能链的可扩展编排：一种联合强化学习方法

网络功能虚拟化 (NFV) 以服务功能链 (SFC) 的形式对各种网络服务的可扩展性和灵活性至关重要，服务功能链 (SFC) 是指以特定顺序链接的一组虚拟网络功能 (VNF)。然而，NFV 性能难以满足网络服务日益增长的需求，主要是由于 SFC 的静态编排。为了解决这个问题，本文通过联合强化学习为支持 NFV 的网络提出了一种新颖的可扩展 SFC 编排（SSCO）方案。SSCO 具有与之前工作不同的三个显着特征：（1）基于联邦学习的框架旨在训练具有时变局部模型探索的全局学习模型，用于可扩展的 SFC 编排，同时避免利益相关者之间的数据共享；(2) SSCO 允许本地客户端和云服务器在每一集的第一个和最后一个 epoch 之间进行参数更新，以确保分布式客户端可以以较低的通信成本进行模型优化；(3) SSCO 引入了一种高效的深度强化学习 (DRL) 方法，利用可用资源和实例化成本的本地学习知识，灵活地将 VNF 映射到网络中。此外，提出了一种基于损失权重的机制来生成和利用重放缓冲区中的参考样本以供将来训练，避免样本的强相关性。从不同工作场景获得的仿真结果表明，与最先进的机制相比，SSCO 可以显着减少放置错误并提高资源利用率，以放置时变 VNF。此外，