Major data centre providers are introducing RDMA-based networks for their tenants, as well as for operating the underlying infrastructure. In comparison to traditional socket-based network stacks, RDMA-based networks offer higher throughput, lower latency and reduced CPU overhead. However, transparent checkpoint and migration operations become much more difficult. The key reason is that the OS is removed from the critical path of communication. As a result, some of the communication state itself resides in the NIC hardware and is no more under the direct control of the OS. This control includes especially the support for virtualisation of communication which is needed for live migration of communication partners. In this paper, we propose the basic principles required to implement a migration-capable RDMA-based network. We recommend some changes at the software level and small changes at the hardware level. As a proof of concept, we integrate the proposed changes into SoftRoCE, an open-source kernel-level implementation of the RoCE protocol. We claim that these changes introduce no runtime overhead when migration does not happen. Finally, we develop a proof-of-concept implementation for migrating containerised applications that use RDMA-based networks.

中文翻译：

---

MigrOS：透明操作系统实时迁移支持容器化 RDMA 应用程序

主要数据中心提供商正在为其租户以及运营底层基础设施引入基于 RDMA 的网络。与传统的基于套接字的网络堆栈相比，基于 RDMA 的网络提供更高的吞吐量、更低的延迟和更少的 CPU 开销。但是，透明的检查点和迁移操作变得更加困难。关键原因是操作系统从通信的关键路径中删除。因此，某些通信状态本身存在于 NIC 硬件中，不再受操作系统的直接控制。这种控制尤其包括对通信伙伴实时迁移所需的通信虚拟化的支持。在本文中，我们提出了实现具有迁移能力的基于 RDMA 的网络所需的基本原则。我们建议在软件级别进行一些更改，并在硬件级别进行一些小的更改。作为概念证明，我们将提议的更改集成到 SoftRoCE 中，这是 RoCE 协议的开源内核级实现。我们声称，当迁移没有发生时，这些更改不会引入运行时开销。最后，我们开发了一个概念验证实现，用于迁移使用基于 RDMA 的网络的容器化应用程序。