The hierarchy theory is the foundation of the modern computer system design. However, the interaction part between different system layers is usually the weak point of the system, which tends to have security flaws. When communicating across the system boundary, failure to enforce the required synchronization in the shared memory can cause data inconsistency of the communication partners. Especially when there is a privilege gap between different boundary sides, such data inconsistency can lead to security vulnerability and sabotage the trust boundary. In this paper, we propose the concept of inter-boundary vulnerability and give the first in-depth study of them. We investigate three typical boundaries in the system that inter-boundary vulnerabilities are prone to occur, including the kernel-user boundary, the hardware-OS boundary, and the VMM-guest OS boundary. Then, based on the investigation of 115 real-world vulnerability cases, we extract four vulnerability types and provide analysis for each type to illustrate the principle. Finally, we discuss the state-of-the-art techniques that are relevant to the detection, prevention, and exploitation of such vulnerabilities, aiming to light the future research on this topic.

层次理论是现代计算机系统设计的基础。但是，不同系统层之间的交互部分通常是系统的薄弱点，这往往会带来安全漏洞。在跨系统边界进行通信时，如果无法在共享内存中强制执行所需的同步，则可能导致通信伙伴的数据不一致。尤其是当不同边界侧之间存在特权间隙时，这种数据不一致会导致安全漏洞并破坏信任边界。在本文中，我们提出了跨边界脆弱性的概念，并对其进行了首次深入研究。我们调查了系统中容易发生跨界漏洞的三个典型边界，包括内核用户边界，硬件操作系统边界，和VMM来宾OS边界。然后，在对115个现实漏洞案例的调查的基础上，我们提取了四种漏洞类型并针对每种类型进行了分析以说明其原理。最后，我们讨论了与检测，预防和利用此类漏洞相关的最新技术，目的是阐明有关此主题的未来研究。