This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The new joint reward and internal rewards mechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes.

中文翻译：

---

用于成本和延迟敏感的虚拟网络功能布局和路由的多代理深度强化学习


本文提出了一种有效且新颖的基于多智能体深度强化学习（MADRL）的方法来解决联合虚拟网络功能（VNF）布局和路由（P＆R）问题，其中同时交付具有差异化需求的多个服务请求。服务请求的差异化需求通过其延迟和成本敏感因素反映出来。我们首先构造一个VNF P&R问题来共同最小化服务延迟和资源消耗成本的加权和，这是NP完全的。然后，联合VNF P&R问题被解耦为两个迭代子任务：放置子任务和路由子任务。每个子任务由多个并发的并行顺序决策过程组成。通过调用深度确定性策略梯度方法和多智能体技术，设计了 MADRL-P&R 框架来执行这两个子任务。提出了新的联合奖励和内部奖励机制来匹配布局布线子任务的目标和约束。我们还提出了基于参数迁移的模型再训练方法来应对不断变化的网络拓扑。经实验证实，所提出的MADRL-P&R框架在服务成本和延迟方面优于其替代方案，并为个性化服务需求提供了更高的灵活性。基于参数迁移的模型再训练方法可以在适度的网络拓扑变化下有效地加速收敛。