Experience-driven networking has emerged as a new and highly effective approach for resource allocation in complex communication networks. Deep Reinforcement Learning (DRL) has been shown to be a useful technique for enabling experience-driven networking. In this paper, we focus on a practical and fundamental problem for experience-driven networking: when network configurations are changed, how to train a new DRL agent to effectively and quickly adapt to the new environment. We present an Actor-Critic-based Transfer learning framework for the Traffic Engineering (TE) problem using policy distillation, which we call ACT-TE. ACT-TE effectively and quickly trains a new DRL agent to solve the TE problem in a new network environment, using both old knowledge (i.e., distilled from the existing agent) and new experience (i.e., newly collected samples). We implement ACT-TE in ns-3, and compare it with commonly-used baselines using packet-level simulations on three representative network topologies: NSFNET, ARPANET and random topology. The extensive simulation results show that 1) The existing well-trained DRL agents do not work well in new network environments; 2) ACT-TE significantly outperforms both two straightforward methods (training from scratch and fine-tuning based on an existing DRL agent) and several widely-used traditional methods in terms of network utility, throughput and delay.

中文翻译：

---

基于Actor-Critic的基于经验的网络迁移学习框架

经验驱动的网络已经成为复杂通信网络中资源分配的一种新的高效方法。深度强化学习（DRL）已被证明是启用体验驱动型网络的有用技术。在本文中，我们重点关注体验驱动型网络的一个实际和基本问题：更改网络配置后，如何训练新的DRL代理以有效，快速地适应新环境。我们使用策略提炼提出了一种基于Actor-Critic的交通学习（Transfer Learning）框架，用于交通工程（TE）问题，我们称之为ACT-TE。ACT-TE使用旧知识（即从现有代理中提取的经验）和新经验（即新收集的样本）来有效，快速地培训新的DRL代理以解决新网络环境中的TE问题。我们在ns-3中实现ACT-TE，并使用三种代表性网络拓扑（NSFNET，ARPANET和随机拓扑）上的数据包级仿真将其与常用基准进行比较。大量的仿真结果表明：1）现有的训练有素的DRL代理在新的网络环境中不能很好地工作；2）ACT-TE在网络实用性，吞吐量和延迟方面均明显优于两种简单的方法（从零开始的培训和基于现有DRL代理的微调）和几种广泛使用的传统方法。