Physical machines in modern datacenters are routinely upgraded due to their maintenance requirements, which involve migrating all the virtual machines they currently host to alternative physical machines. For this kind of datacenter upgrades, it is critical to minimize the time it takes to upgrade all the physical machines in the datacenter, so as to reduce disruptions to cloud services. To minimize the upgrade time, it is essential to carefully schedule the migration of virtual machines on each physical machine during its upgrade, without violating any constraints imposed by virtual machines that are currently running. Rather than resorting to heuristic algorithms as existing work, we propose a new scheduler, Raven, that uses an experience-driven approach with deep reinforcement learning to schedule the virtual machine migration. With our design of the state space, action space and reward function, Raven trains a fully-connected neural network using the cross-entropy method to approximate the policy of choosing a destination physical machine for each virtual machine before its migration. We compare Raven with state-of-the-art algorithms in the literature, and our results show that Raven can effectively shorten the time to complete the datacenter upgrade under different datacenter settings.

由于维护要求，现代数据中心中的物理机会定期进行升级，这涉及将其当前托管的所有虚拟机迁移到备用物理机。对于此类数据中心升级，至关重要的是最大程度地减少升级数据中心中所有物理机所需的时间，以减少对云服务的破坏。为了最大程度地减少升级时间，至关重要的是在升级过程中仔细计划每个物理机上虚拟机的迁移，而不会违反当前正在运行的虚拟机施加的任何约束。与其采用启发式算法作为现有工作，不如提出一个新的调度程序Raven，它使用经验驱动的方法进行深度强化学习，以安排虚拟机迁移。通过我们对状态空间，动作空间和奖励功能的设计，Raven使用交叉熵方法训练了完全连接的神经网络，以近似估算在迁移之前为每个虚拟机选择目标物理机的策略。我们在文献中将Raven与最先进的算法进行了比较，结果表明Raven可以有效地缩短在不同数据中心设置下完成数据中心升级的时间。