In delay-sensitive industrial internet of things (IIoT) applications, the age of information (AoI) is employed to characterize the freshness of information. Meanwhile, the emerging network function virtualization provides flexibility and agility for service providers to deliver a given network service using a sequence of virtual network functions (VNFs). However, suitable VNF placement and scheduling in these schemes is NP-hard and finding a globally optimal solution by traditional approaches is complex. Recently, deep reinforcement learning (DRL) has appeared as a viable way to solve such problems. In this paper, we first utilize single agent low-complex compound action actor-critic RL to cover both discrete and continuous actions and jointly minimize VNF cost and AoI in terms of network resources under end-to-end Quality of Service constraints. To surmount the single-agent capacity limitation for learning, we then extend our solution to a multi-agent DRL scheme in which agents collaborate with each other. Simulation results demonstrate that single-agent schemes significantly outperform the greedy algorithm in terms of average network cost and AoI. Moreover, multi-agent solution decreases the average cost by dividing the tasks between the agents. However, it needs more iterations to be learned due to the requirement on the agents’ collaboration.

中文翻译：

---

使用深度强化学习在工业物联网中进行信息感知 VNF 调度的时代

在对延迟敏感的工业物联网 (IIoT) 应用中，信息时代 (AoI) 用于表征信息的新鲜度。同时，新兴的网络功能虚拟化为服务提供商使用一系列虚拟网络功能 (VNF) 提供给定网络服务提供了灵活性和敏捷性。然而，这些方案中合适的 VNF 放置和调度是 NP-hard 的，并且通过传统方法寻找全局最优解是复杂的。最近，深度强化学习 (DRL) 已成为解决此类问题的可行方法。在本文中，我们首先利用单代理低复杂复合动作 actor-critic RL 来覆盖离散和连续动作，并在端到端服务质量约束下共同最小化网络资源方面的 VNF 成本和 AoI。为了克服单代理学习能力的限制，我们将我们的解决方案扩展到多代理 DRL 方案，其中代理相互协作。仿真结果表明，单代理方案在平均网络成本和 AoI 方面明显优于贪心算法。此外，多代理解决方案通过在代理之间划分任务来降低平均成本。然而，由于对代理协作的要求，它需要更多的迭代来学习。多代理解决方案通过在代理之间划分任务来降低平均成本。然而，由于对代理协作的要求，它需要更多的迭代来学习。多代理解决方案通过在代理之间划分任务来降低平均成本。然而，由于对代理协作的要求，它需要更多的迭代来学习。