Abstract

In recent years, increased attention is being given to software quality assurance and protection. With considerable verification and protection schemes proposed and deployed, today’s software unfortunately still fails to be protected from cyberattacks, especially in the presence of insecure organization of heap metadata. In this paper, we aim to explore whether heap metadata could be corrupted and exploited by cyberattackers, in an attempt to assess the exploitability of vulnerabilities and ensure software quality. To this end, we propose RELAY, a software testing framework to simulate human exploitation behavior for metadata corruption at the machine level. RELAY employs the heap layout serialization method to construct exploit patterns from human expertise and decomposes complex exploit-solving problems into a series of intermediate state-solving subproblems. With the heap layout procedural method, RELAY makes use of the fewer resources consumed to solve a layout problem according to the exploit pattern, activates the intermediate state, and generates the final exploit. Additionally, RELAY can be easily extended and can continuously assimilate human knowledge to enhance its ability for exploitability evaluation. Using 20 CTF&RHG programs, we then demonstrate that RELAY has the ability to evaluate the exploitability of metadata corruption vulnerabilities and works more efficiently compared with other state-of-the-art automated tools.

1. Introduction

Since cyberattacks have severely damaged the quality of software [1, 2], considerable defensive schemes have been taken to protect software security [3, 4]. In recent years, due to the deployment of mitigation mechanisms for stack-based vulnerabilities and compiler-level inspections, stack exploits have become increasingly more difficult to execute against hardened programs [5, 6]. However, there are still a large number of heap-based vulnerabilities in software written in insecure programming languages such as C and C++ [7–9]. These heap vulnerabilities increase the possibility of malicious attacks and pose a serious threat to software security [10].

The threat from hacking mainly comes from heap metadata corruption [11–13]. To improve performance, the heap allocator usually places dynamically allocated user data in the same memory area as the metadata that control the behavior of the allocator [14]. These metadata are unprotected, and vulnerabilities related to processing user data may overwrite metadata, which causes subsequent operations of the heap allocator to violate security assumptions [15]. Since this is an inherent defect of the heap allocator, it is not limited to any specific applications. Any program that employs the heap allocator may trigger such metadata attacks. Although developers have introduced enhanced mechanisms to protect heap metadata [16, 17], such as the 2017 patch, attackers can easily bypass the protection by slightly modifying their behavior [18]. Because of this adaptation, there have still been widespread attacks on metadata [19].

AEG (automatic exploit generation) was introduced to quickly generate exploits and improve the ability to assess the exploitability of software vulnerabilities [20–23]. In recent years, heap-based AEGs have improved, and some simple exploitations for user-data corruption can be processed [24, 25]. However, the exploitation of metadata requires a high degree of skill, and so this process is usually combined with human expertise [26]. Existing AEGs do not integrate the required high-level expertise well, making it difficult to complete metadata attacks. Specifically, we can subdivide the problems of AEGs in metadata attacks into the following two aspects.

1.1. Coarse-Grained Heap Layout Arrangement

The existing AEG methods lack the expertise of the intermediate exploitation process, and it uses a saltatory strategy to complete the migration of memory state (MMS). For example, Revery [24] jumps directly from the panic state to the memory state including arbitrary address writing (AAW). However, in most cases, the exploitation of heap vulnerabilities, especially metadata corruption, needs to be closely combined with human expertise, and it is necessary to traverse intermediate memory states in a step-by-step strategy.

1.2. Blindness for Driving MMS

The exploitation in the kernel [27] and the interpreter [28] can directly trigger a specific heap operation through the exposed interface to construct the target memory state. Heaphopper [29] and Archeap [11] search for exploitation primitives by connecting the heap allocator to the driver program. They analyze the exploitability of the metadata in the allocator, without considering exploit generation of real programs embedding in the allocator. However, driving the MMS for user-level applications is still a nontrivial problem. It is necessary to satisfy the constraints of both path and state reachability and solve for the corresponding input. Existing AEG drives MMS through a blind search, and it is difficult to solve the above constraints.

To solve the above problems, we propose a new solution called RELAY, a software testing framework that simulates human exploitation behavior at the machine level to facilitate exploitability evaluation for software vulnerabilities. RELAY employs exploit patterns converted from human expertise to guide MMS, and it can generate exploits for metadata corruption.

We propose a serialization method for heap layout, which transforms the human expertise of exploitation into a sequence of memory states that the machine can understand and solve. The process of a metadata attack can be viewed as a combination of memory states following in a specific order, and the attack will fail even if there is only a slight change in this order. These memory states and their sorting processes are highly dependent on artificial knowledge such as Unlink [30]/Hof [31] exploit methods. Based on this information, RELAY employs a predicate-based memory characterization method to depict the critical memory state of exploitation and build a relationship chain of state migration called an exploit pattern. Through building an exploit pattern, we can refine the memory status of the heap layout, making the fuzzy and disordered heap layout more ordered. At the same time, it also converts a single difficult migration problem of memory state into multiple simple problems, which reduce the difficulty in solving constraints.

RELAY builds exploit pattern at the logical level and then needs to construct a state migration path at the machine level and drives programs to complete the MMS. Based on the heap layout serialization method, we employ a novel heap layout procedural method to drive the MMS. Specifically, with the memory operation primitive (MOP) graph and heap allocator model, RELAY first solves a set of heap operation sequences, which satisfy numerical and structural constraints to build a , and then approximates the target memory state by exploring paths, generating finally corresponding inputs. With the heap layout procedural method, RELAY can effectively drive the process of exploitation at a small cost.

In summary, this paper makes the following contributions:(i)We propose a new solution RELAY, a software testing framework for simulating human exploitation behavior at the machine level, which can evaluate exploitability for metadata corruption vulnerabilities. In particular, RELAY can be easily extended and can continuously assimilate human expertise to enhance its ability for exploitability evaluation.(ii)We propose a serialization method of heap layout, which transforms the high-level expertise of exploitation into an orderly memory state combination that the machine can understand and solve.(iii)We propose a novel procedural method of heap layout to achieve MMS driving and activate exploit patterns at the machine level.(iv)We implement a prototype of RELAY and demonstrate its effectiveness with CTF (Capture the Flag) and RHG (Robot Hacking Game) programs.

The rest of this paper is organized as follows: Section 2 describes research motivation and presents the overview of RELAY; Section 3 proposes the heap layout serialization method to construct exploit patterns; Section 4 proposes the heap layout procedural method to manipulate the heap layout according to the exploit patterns; in Section 5, we explain the implementation of RELAY and give an experimental evaluation of the system; Section 6 summarizes the work that is most relevant to ours; finally, we summarize the work and discuss the potential research orientation in the future in Section 7.

2. Motivation Example

In this section, we display exploitation for a common vulnerability to illustrate the challenges faced by existing AEGs and introduce our solution.

2.1. Metadata Exploitation Example

The heap allocator usually maintains metadata (such as size, presize, fd, and bk in ptmalloc [32]) to manage heap blocks and control heap behavior. Since the heap allocator does not fully protect the metadata, if developers do not pay attention to the safety rules when programming and invoke the heap allocator without due consideration, a hacker will be able to break the software security by corrupting the metadata. Due to the above reasons, metadata attacks are widely used throughout the real world.

Security analysts have further summarized some attack methods against metadata corruption, such as FastBin attack [33], Unlink, Hof, and chunk overlap&FastBin attack [34]. For FastBin attack, we need to modify the fd pointer of the released heap block and fake a FastBin heap block and then take down the forged heap block from the FastBin linked list through multiple allocations, resulting in arbitrary address writing (AAW). For Unlink, we need to modify the fd and bk pointers of the freed chunk and trigger the unlink operation by merging heap blocks for an AAW. For Hof, we need to modify the size of TopChunk, calculate the distance between target address and current TopChunk address, and trigger the allocation with a larger size. Finally, the system will return the target memory block to users as an allocated chunk. For chunk overlap&FastBin attack, we need to reuse the common memory between two heap blocks and thus tamper with fd pointer to continue with FastBin attack (more specific details will be described later).

It is complicated to implement these exploitation methods, including precise calculation and organized layout arrangement. But as shown above, the metadata may be damaged easily by heap overflow, Use-After-Free (UAF), off-by-one [35], and other vulnerabilities due to inadequate protection. So these exploitation methods are widely used together with the frequent metadata attacks. Therefore, the critical issue we need to solve is how to combine these popular but complicated exploitation methods to strengthen the ability of the machine to evaluate the exploitability of vulnerabilities automatically and thus maintain the software security.

Next, we will present the existing problems of AEG through case analysis and explore the solutions. We use the common off-by-one vulnerability as an example, which is a vulnerability often used to destroy size metadata, combined with the FastBin attack and unlink methods to cause arbitrary code execution. Figure 1 shows a typical exploitation process of an off-by-one vulnerability. The memory operations are required at each step, and the corresponding critical memory states are shown as follows:(1)By overflowing one byte, the last byte of the size metadata of allocated chunk 2 can be controllable.(2)By freeing chunk 2 and then reoccupying the position, chunk 2 can be extended and overlapped with chunk 3.(3)By releasing chunk 3 and then controlling its contents with the extended chunk 2, fd pointer of chunk 3 can be controllable.(4)By allocating chunks multiple times, memory 4 (callable memory) pointed by the controlled fd pointer can be allocated. We achieve an arbitrary address allocation in this step.(5)By writing data to chunk 4, this callable memory is controllable.(6)By invoking memory 4, the program reaches the state of control-flow hijacking.

Figure 1 

Typical exploitation process for off-by-one vulnerability, the number before the chunk represents its size.

2.2. Challenge

The above process can represent most metadata attacks, and they usually need to complete the migration between multiple states by multiple steps. The existing AEG method cannot refine these intermediate memory states. It is difficult to directly search for primitives such as control-flow hijacking and arbitrary address writing (AAW) from the panic state. Besides, it can be seen that the exploitation of metadata requires an in-depth understanding of the internal mechanisms of the heap allocator; the definition and ordering of intermediate states during exploitation is also based on human expertise. Existing AEGs do not use human knowledge well, and it cannot simulate the behavior of artificial exploitation at the machine level.

In addition, for each step, the program needs to trigger a specific operation such as write, malloc, and free to enter an intermediate state. After these operations are triggered, the correction of the parameters or write content corresponds to the solving of numerical constraints, which can be completed by the symbolic execution tool [36, 37]. How to trigger these heap operations corresponds to the solving of structural constraints. We need to drive the program to enter a specific path branch at a specific time to solve these structural constraints, which is usually done by exploring the path with fuzzing tools [38, 39]. Obviously, it is not easy to complete the oriented path coverage. However, the MMS usually requires triggering these operations frequently. For example, step 4 may require triggering malloc repeatedly to remove chunk 4 from the free list, which often brings with it greater complexity and consumption. Existing AEG methods have a limited ability to solve such structural constraints, and exploring paths blindly is usually inefficient.

2.3. Solution

Figure 2 shows RELAY’s framework. In the serialization part of the heap layout, RELAY transforms human expertise of exploitation into an exploit pattern library. The procedural part of the heap layout mainly consists of two modules: the migration path construction (MPC) module and the migration path drive (MPD) module. First, this part takes PoC as input, analyzes the current memory state, and obtains the corresponding state migration pair from the exploit pattern library. Then, the MPC builds the that satisfies the structural constraints in , and then, the MPD is responsible for driving the in the real environment and generates the corresponding input. The two iterate through the intermediate states, solve the constraints that satisfy path reachability and state reachability, and generate control-flow hijacking inputs. Finally, the exploit generation module outputs exploits to complete the specific attack.

Figure 2 

Overview of RELAY’s frame. PoC represents proof-of-concept.

2.3.1. Heap Layout Serialization

First, this module employs the predicate-based memory characterization method to describe . Then, by combining the advanced knowledge observed in human exploitation behavior, it assembles the to build an exploit pattern for a typical exploitation process. This exploit pattern is a migration chain composed of a series of intermediate states in the exploitation process. For example, the migration from the state of controllable fd pointer in step 3 to the state of arbitrary address allocation in step 4 can form a section of the exploit pattern.

2.3.2. Migration Path Construction Module (MPC)

First, this part obtains a control-flow graph (CFG) through static analysis and based on this builds a memory operation primitive (MOP) graph from which the memory operation groups are extracted (MOGs). Then, we establish a heap allocator model to infer the memory state corresponding to the MOG and build the by comparing the target memory states of the migration pair. Finally, we utilize the triggering probability of the path to solve the with the smallest driving cost. For example, for a migration demand from state 3 to state 4, the can be solved as MMM (M represents malloc). This module can interact with an MPD.

2.3.3. Migration Path Driver Module (MPD)

After receiving the (A for allocation) from the MPC module, the MPD drives the with fuzzing at the machine level and attempts to activate state 4. If state 4 is not activated after driving, the MPD will reselect the constructed by the MPC and launch it again for the current migration pair. If activated, the MPD will retain the input and switch to the next migration pair 4-5. By repeatedly executing the MPC and the MPD, RELAY will activate all the intermediate states at the machine level.

2.3.4. Exploit Generation Modules

This module receives test cases that can hijack PC (program counter) and perform symbolic constraint solving on the PC and the related controllable memory space to generate the final exploits. In terms of AEG for EIP-hijacking vulnerabilities such as Mayhem [23] and CRAX [40], the critical part of these schemes is that, after hijacking EIP, the control flow will be guided to the target position by symbolic execution, and thus, the PoC input is transformed into EXP input. We reuse their work for our exploit generation module. Specifically, if it is found that the EIP register contains a symbolic value, and we will arrange the shellcode into memory with permission W + X and then add a symbol constraint to make the symbol value point to the shellcode location, redirecting control flow to that location and generating EXP for arbitrary code execution. Since this is a previous work, we will not repeat it in this paper.

3. Serialization Method of Heap Layout

The existing AEGs perform badly in refining the heap layout and often “jumps” during the processing of the heap layout. For example, Revery [24] jumps directly from the state of triggering a vulnerability to the state of hijacking the control flow. The limitation is that they adopt a random exploration strategy in the heap layout process, and they are not guided sufficiently by human expertise. Therefore, we need to explore a method that can guide the heap layout process, apply this expertise at the machine level, and make the machine solve the complex problems of metadata attacks using the way of human thinking.

To this end, we propose a serialization method of heap layout, combined with the way humans think about crafting exploits, to decompose the problem of solving the highly complex process of the heap layout into multiple subproblems with low complexity. First of all, we employ a machine-understandable memory descriptive language to represent the memory attributes, the data, and structural relationships between memory objects, to ensure accurate characterization of intermediate states. Then, we refine the “fragmented knowledge” of manual exploitation to model the migration pair and orderly combine the migration pair according to typical exploit methods. Finally, we can build an exploit pattern that can be utilized to guide the heap layout process.

3.1. Predicate-Based Memory Characterization Method

This section employs a formal method of describing memory states which can be easily translated into programming languages and can be implemented at the machine level. Specifically, this method characterizes the memory state through three elements: memory objects, object attributes, and relationships between objects. The relationships between objects include numerical and structural relations. This section first clarifies the definition of memory objects. Then, it describes the properties of these memory objects and the numerical and structural relationships between these objects based on predicates.

3.1.1. Extended Representation of Memory Objects

Memory objects are usually represented by a tuple , where is the address of the memory unit and is the width. For example, indicates the memory unit of consecutive 4 bytes located in the address . We represent the structured memory objects such as the heap block on the basis of the tuple. These memory objects are discussed below in the 32 bit architecture.

Although the structure objects in the heap memory such as heap blocks and free linked lists are complicated, the basic processed objects are heap blocks. Therefore, the structured description of heap memory is also based on heap blocks. The heap block is divided into the allocated state and the free-ed state, and so we utilize different data structures to characterize them.

The heap block in the allocated state can be formalized as , where represents the address of the chunk; represents the size of the physical neighbouring chunk at the low address; represents the size of the chunk; represents the data area of the chunk; and represents whether it is the main allocated area, whether it is mapped memory, and whether the physical neighbouring chunk at a low address is in the allocated state. All of the elements in the formal representation of are -type memory objects ( can be regarded as a 4 byte memory block storing the chunk address). The heap block in the free-ed state adds a forward pointer and a backward pointer (both are -type memory objects) on the basis of , that is, .

Furthermore, the free list can be defined as , where is a free-ed chunk; is of numeric type and represents the length of the free list ; is of Boolean type and represents whether the linked list is for random access; and and are only for nonrandom access linked lists, and they are of numeric types, respectively, representing the direction of inserting and deleting nodes of the linked list (0 represents the direction from the head of the linked list, and 1 represents the direction from the tail of linked list). According to whether and are the same, the free list for nonrandom access can be divided into a one-way operation list (such as FastBin in ptmalloc) or a two-way operation list (such as SmallBin).

Based on the heap block and free linked list, the heap memory can be formally described as , where is the set of allocated chunks in the entire memory space (A contains not only the heap blocks allocated in the heap memory but also the heap blocks that attackers may forge); stands for FastBin, SmallBin, and LargeBin, respectively. stands for UnsortedBin; stands for TopChunk; stands for Last Remainder (the structure of is similar to a chunk in the free-ed state, so we will not go into details).

With the representation method of structured heap memory, we can complete the fine-grained description of the heap memory state. For example, depicts the situation where the flag of allocated memory has overflowed (we use to indicate that are controlled by external input, which is strictly defined later), and depicts that the size field of the TopChunk has overflowed. It can be seen from the former expression that an accurate description of the memory state must be based on the accurate definition of the attributes of the memory objects (such as controllable ) and relationships between memory objects (such as ), so we will separately introduce their definition in the following sections.

3.1.2. Attribute Predicate Definition

We have extended conventional memory attributes to support complex attribute descriptions of memory units. For each memory unit, we introduced three additional attributes besides the readable, writable, and executable attributes: controllable, callable, and chunk association, as shown in Table 1.

The following three examples explain these three new attributes. It can be expressed as if TopChunk’s size field is overflowed. indicates that the data stored in consecutive 4 bytes of memory unit located at address can be directly or indirectly migrated to the PC register (for example, the data of the memory unit where the return address is located will be written to the PC register when the function returns, and the data in the GOT table will be migrated to the PC if invoking the corresponding function). indicates that the data stored in the 4 consecutive bytes of memory unit located at address can be used as a heap block pointer for allocation, release, and writing operations.

3.1.3. Relation Predicate Definition

This section analyzes the relationships between memory objects from two aspects: the numerical relationship and the structural relationship . The numerical relationship is used to indicate the relationship of memory objects at the numerical level. Common numerical relationships are shown in Table 2. For example, it can be expressed as if the size of the heap block is equal to that of the heap block . If the size of the heap block meets the requirement of the free linked list , it can be expressed as . Besides representing the numerical relationship between objects, can also be used to express the relationship between objects and values; for example, indicates that the size of the chunk is 64 bytes.

The structural relationship is used to indicate the relationship between memory objects at the structural level. The 9 typical structural relationships are shown in Table 3. For example, it can be expressed as if the free-ed chunk c1 belongs to the FastBin list. For the unstructured memory object mem that stores the pointer pointing to the data area of chunk , it can be expressed as . An equivalence relation is used to indicate that the elements that constitute two objects are the same and the memory area occupied is completely the same; that is, the two objects are essentially the same object. An above relationship and below relationship are used to describe the positional relationship of memory objects in physical space. The forward/backward relationship is similar to the relationship. The only difference is that the is used to indicate that the adjacency relationship is higher or lower in the current structured object level. For example, indicates that the chunk next to the chunk in the direction of high address is . is used to indicate that there is a common memory object mem between and .

3.2. Exploit Pattern Construction

Exploitation can be expressed as the migration process of a special memory state in the program state space. Therefore, we employ state-space notation to describe the exploit process. State-space notation can be defined as quaternion , where is the set of possible states, is the initial state of the program, is the set of possible operators, and is the target state. This section describes the state migration process at the model level by constructing the exploit pattern. The definition of the operator and the realization of the state migration at the machine level are described in the next section.

The experience of artificial exploitation consists of a series of pieces of “fragmented knowledge;” for example, an “AAW” may result in “control-flow hijacking” and “controllable format string” may result in “AAW”. The process of manual exploitation essentially organizes these pieces of “fragmented knowledge” in a specific order. For example, the process of implementing arbitrary code execution using format string vulnerabilities can be expressed as “format string controllable” ⟶ “AAW” ⟶ “control flow hijacking”. How to represent this “fragmented knowledge” and how to organize it is the key to applying human expertise to the state migration process.

3.2.1. State Migration Pair

We employ state migration pairs to represent “fragmented knowledge.” can be expressed as , where and are the memory states portrayed by predicates and represents the migration path formed by the orderly combination of several operations. Table 4 lists several typical state migration pairs related to heap exploits under 32 bit systems, where represents the integer value at address , and represents adding the objects and (or numerical value ) on the numerical level.

Table 4 does not need to ensure the completeness of the characterization. Firstly, the state migration pairs in this table can be continuously updated according to human experience; and secondly, the source and destination states of the migration pair do not need to be fully characterized for all memory and register objects but only focus on the critical memory objects, structural relationships, and numerical relationships based on the human experience.

3.2.2. Exploit Pattern

The artificial exploit methods can be regarded as a combination of “fragmented knowledge” in a specific order. We combine in the same order to build an exploit pattern to simulate human exploit methods.

It should be noted that, before constructing the exploit pattern, we need to backpropagate the constraints collected in the migration pairs to ensure the effectiveness of the exploit pattern. For example, migration pair requires that the memory at p in the original state has a callable attribute, so combination requires backpropagation to the source state of migration pairs, i.e., . In order to facilitate expression, we distinguish memory objects in different states through the subscript (for instance, and represent the same memory area in different states), so the combination can be expressed as the following formula after the constraint propagation is completed:

We have established corresponding exploit patterns for common exploit methods. FastBin attack can be expressed as , and the exploit pattern is as follows:

Unlink can be expressed as , and the exploit pattern is as follows:

Hof can be expressed as, and the exploit pattern is as follows:

The example in Section 2.1 shows a typical exploit method for off-by-one vulnerabilities, and we can call it chunk overlap&FastBin attack, which can be expressed as :

3.2.3. Guidance of Exploit Generation

It can be seen that the exploit pattern can characterize the migration relationship of the intermediate state during the exploitation process, including the data relationship and structural relationship between the memory objects. In order to further guide exploit generation, we divide the predicates used to characterize the memory state into two categories: numerically dependent predicates and structurally dependent predicates. The data dependency predicates refer to , and other predicates that reflect the numerical relationship between memory objects; structure dependency predicates mainly include the extended memory attribute predicate (, and ) and , and other predicates that reflect the structural relationship between memory objects. These two types of predicates correspond to form two different types of constraints, namely, numerical constraints and structural constraints . The exploit pattern can transform the problem of automatic generation of heap exploits into the problem of solving numerical constraints and structural constraints .

We employed the exploit pattern as an example to explain how to guide the exploit generation by solving these two types of constraints. The exploit consists of 4 steps corresponding to 4 states in :(1)Now, the fd pointer of chunk in the FastBin list is controllable, which has satisfied the following structural constraints: But, we have not made the .a and .a equal, which means the following numerical constraints have not been satisfied: In order to solve , we should calculate the address of . The data area of needs to cover , and needs to be callable, such as malloc_hook. So, we can update the numerical constraints as follows: We solve it by using a constraint solver.(2)The following numerical constraints have been satisfied: The following structural constraints require making become an allocated chunk: If is located in the Nth position from the head end of FastBin linked list (namely, there are N-1 heap blocks before ), we need, according to the FIFO (first in last out) mechanism of FastBin, to trigger the allocation for consecutive N times to take out , and the size of the allocation needs to satisfy the requirements of the FastBin list.(3)Now that numerical constraints have been satisfied, we only need to solve the following structural constraints: is malloc_hook, which satisfies, so we just need to cover the malloc_hook to satisfy .(4)After controlling the malloc_hook, we need to invoke malloc to transfer the data stored in the malloc_hook to the PC register, which implements C(PC).

4. Procedural Method of Heap Layout

The serialization method of heap layout divides the exploitation process into an ordered set of multiple intermediate states at the logical level by constructing the exploit pattern and indicates the direction for the heap layout.

We also need to activate the intermediate state at the machine level and implement the migration process of the heap layout.

4.1. Framework

With the exploit pattern, we reduce the complexity of traversing the state space of the program, but the extremely large path space also places a significant burden on exploitation. It is often inefficient to drive the program to the target state by randomly exploring the program path. Taking the exploitation process for off-by-one as an example, in moving from state 3 to state 4, a certain number of heap allocation operations are required and the size of the allocated heap block must meet the requirements of the linked list. Blindly traversing the program path will result in a random combination of memory operations and their parameters, which is very inefficient.

Therefore, we divide the heap layout process into two stages: migration path construction (MPC) and migration path driving (MPD). The MPC phase constructs a set of state migration paths according to the exploit pattern, and the MPD phase attempts to drive the program along the . By building the to guide the state migration of the program, we can reduce the complexity of traversing the path space and improve the driving efficiency.

The overall design of the procedural method of heap layout is shown in Figure 3. Through the heap layout serialization module, we build the exploit pattern. The program state analysis module analyzes the current program state and finds which source state matches with it. We can decompose into numerical constraints and structural constraint (as explained in the previous section). In each round of processing an , the numerical constraint solving module first corrects the input related to . Next, the MPC generates a set according to the structural constraint . Then, the MPD performs searches according to the and drives the state migration of the program. In the loop, we first adopt an optimization strategy. For the migration path set , we give priority to the one with the smallest driving cost and reduce the complexity of the driving migration path as much as possible. Secondly, we add a feedback mechanism. If the new state is not matched with the target state after the path is driven (the matching method will be explained later), we give up the intermediate test case and restore to the original state of the program, remove the current migration path from the set, and then start a new round of retrying to drive. If matched, the program state analysis will pass the next to the MPC and the MPD. By repeatedly solving and , RELAY can complete the whole MSM, activate all intermediate states, and finally generate the hijacking PC input.

Figure 3 

Design of the procedural part for RELAY.

Since the existing symbolic execution technology can solve the numerical constraints in the exploit pattern [41], we will not repeat the process of solving but explain how to automatically solve the structural constraints in the exploit pattern.

4.2. Migration Path Construction (MPC)

The MPC constructs migration paths according to structural constraints. First, the MPC constructs the memory operation primitive (MOP) graph through static analysis and extracts the memory operation groups (MOGs) from it. Then, it employs the heap allocator model to solve the from the MOG according to the structural constraints and selects the k paths with the smallest driving cost (the relevant definition will be described later) to build the optimized set . In the MOP graph section, we give a formal definition of and explain why structural constraints can be solved by using an ordered set of MOPs. Based on this, we construct a MOP graph and introduce a method to extract the heap operation groups from it. The heap allocator model is relatively independent and easy to expand to support different heap allocators. It establishes a mapping relationship between heap operations and memory states, and it is used to examine the effect of heap operations. In order to find the solution, it takes the structural constraints as input and employs the MOP graph to obtain all possible groups of heap operations. Then, it utilizes the heap allocator model to solve the group of heap operations that can meet the structural constraints as and finally takes the k paths with the smallest driving cost to form a output set .

4.2.1. MOP Graph

The structural constraint cannot be solved by calculation tools such as constraint solvers [42] and can only be solved by effectively combining basic operations on the heap memory structure. We abstract the typical heap memory structure operations into five basic memory operation primitives: allocation (A), deallocation (F), memory write (W), memory read (R), and memory call (V), which will be explained separately below.

The allocation primitives are built on the granularity of functions, such as malloc and realloc in ptmalloc, where realloc can be regarded as an allocation or a deallocation. The deallocation primitive is also built on function granularity, typically free in ptmalloc. Writing and reading primitives can be completed by an instruction (such as mov [addr] and val), basic block, or external function (such as read function). In order to reduce the analysis complexity, we define the memory write/read built on basic block granularity. The calling primitive is built on the calling method. For a memory unit with callable attributes, if the target calling method is empty, it means that the data for the memory unit (such as the return address when the function returns) can be automatically transferred to the PC register implicitly; if the target calling method is not empty, the calling primitive is generally triggered by invoking the data in the memory unit in the form of a function pointer (such as invoking malloc_hook when executing malloc).

Based on the above MOPs, we can formally define the and convert the problem of solving structural constraints into a problem of solving an ordered set of MOPs. The migration path between states refers to an ordered set of several MOPs expressed as , . In the case of satisfying the numerical constraint , after program executes these operations, the memory state satisfies the structural constraint relationship ; that is, .

In order to construct the , we need to select a group of MOPs that can satisfy the structural constraints from the program path. We must first establish an understanding of the program path from which to obtain all possible groups of MOPs, so we proposed the concept of MOP graph. The MOP graph represents all of the MOPs that may be traversed during the execution of a program. It represents all possible types and sequences of all MOPs in the form of a graph. As shown in Figure 4(b), each node in the figure represents a MOP, and the directed edge indicates the possible transfer direction of the MOP.

(a)

(b)

(c)

(a)
(b)
(c)

Figure 4 

Constructing the MOP graph and extracting the MOG from it. The circles and lines represent the nodes and edges of the MOP graph, respectively. The blue line represents the strengthened edge. The dotted line represents the traversal path . The number represents the weight of the edge called driving cost (explained later). (a) Mark nodes and edges in CFG; (b) build MOP graph; (c) extract MOG from MOP graph.

The trigger of a MOP is bound to the program path, and the program path can be reflected by CFG, so we employ CFG to extract the MOP graph. The extraction process is shown in Figure 4(a). First, we utilize a static analysis tool to obtain a complete CFG which includes the basic block address, instruction information, and the jump relationship between the basic blocks. Next, we mark the location of the MOP on the CFG as a new node. For the MOP of function granularity, we mark them in the basic block according to the function invocation address. For the MOP of basic block granularity, we mark the corresponding first address of the basic block. We use the tuple to represent the node, where indicates the type of MOP (5 types in total, ) and represents the marked address.

After marking the new node, we establish a new directed edge according to the connection relationship of the new node in the CFG. Specifically, if there is a connectivity path between the basic blocks where the two new nodes ( and ) are located and there are no other new nodes in , then we can build a directed edge . For example, the basic blocks where are located have a connection relationship shown in Figure 4(a), and therefore, a directed edge can be established. The directed edge indicates that the program may perform the operation after the operation, so we can append the operation after the operation when extracting the MOG.

In particular, for the directed edges , if has only one new successor node and any path in CFG from passes through the node, we call a strengthened directed edge. For example, since there is a connection relationship in Figure 4(a), satisfying the above relationship, a strengthened directed edge can be established. The strengthened directed edge means that the program will inevitably invoke the operation after the operation, so we must append the operation after the operation when extracting the MOG.

After traversing all of the marked new nodes, we keep the new nodes and the new directed edges, remove the irrelevant basic blocks, and jump relationships, to complete building the MOP graph, as shown in Figure 4(b).

From the MOP graph, we can explore the path with the graph traversal algorithm and extract all possible MOGs from it. The traversal method is as follows: first, we must determine the starting point of the traversal. Since the is a stitching of the paths in the test case, we need to ensure that the selected MOG can be appended to the original operation. This requires us to determine the starting point for the traversal according to the corresponding program state point in the test case. We record the memory operation execution trace of the program in the test case and select the last memory operation . We then find the node in the MOP graph whose address matches with ’s, which means we find out the current program memory operation in the graph. So, we use this node as the starting point for exploring to ensure that the constructed can be spliced into the current program path. After this, we need to determine the traversal method. Since we need to solve while exploring (explained later), which requires improving the efficiency of exploring shallow paths, we adopt a breadth-first traversal method along the directional edge (consider repeatedly traversing nodes in the ring).

As shown in Figure 4(c), we collect MOPs corresponding to the nodes on each traversed path as a group . It should be noted that if a traversed node has a strengthened directed edge , the node cannot be used as the endpoint of the traversal path . The path must be extended backward to add ’s successor node to the group . Because some memory operations are accompanied by noise, when program triggers these operations, it will inevitably trigger additional memory operations. It is reflected in the MOP graph that these operations have a strengthened edge. We can take the noise problem into consideration by adding special treatment of the strengthened edge, and so the operation group extracted is more reasonable. On the contrary, for general directed edges, there is not such a strong correlation between the front and back operations, so they can be randomly combined without special treatment.

4.2.2. Heap Allocator Model

We have already explained above that, on blindly driving the program, traversing the path space to meet structural constraints is often inefficient. We hope to find a specific path in advance and then drive the program along with it. By reducing the complexity of the path space, we can improve the efficiency of the drive. In order to find the , we first extracted a series of combinations of operations from the MOP graph, and then, we needed to find a group that satisfied the structural constraints. To this end, we established a heap allocator model to infer memory operations, observed the memory state after performing memory operations, and selected an operation group that allows the memory state to satisfy structural constraints.

It should be noted that the model is not only meant to replace the work of the heap allocator, such as managing the heap memory to store data, but only to imitate the characteristics of the heap allocator and analyze the relationship between memory operation and memory state. So we only established the mapping relationship between memory operations and memory state at the logical level and did not actually allocate or deallocate memory at the machine level. In addition, because we only focus on the critical objects associated with structural constraints in the heap memory, there is no need to fully characterize all heap memory.

The essence of the heap allocator is to input the memory operation and output a memory state , so the key to building a heap allocator model is to build a mapping relationship between the memory operation and the memory state. After the program invokes a memory operation, the memory changes can be expressed as the relationship . The output state is determined by the operation and input state . Therefore, a binary functional relationship can be established, and our goal is to parse this binary function. In order to express binary functions in an enumerable form, we need to employ finite symbolic expressions to represent infinite specific values of memory states and memory operations. To this end, we use the extended representation of the memory object to express the memory state and then use the unified operation symbol to express the memory operation to convert the binary function into an exhaustible segmented form.

Taking ptmalloc as an example, we show how to build a heap allocator model and express it as a binary mapping relationship. For convenience, the size represents the aligned parameter of allocation, represents the newly allocated chunk, and represents the released chunk. indicates that the is within the range of the linked list. represents that is the smallest of all chunks in the linked list whose size is larger than . represents the split of into and . indicates the merge of into . represents moving chunks into the linked list from UnsortedBins, and represents moving the split chunk into UnsortedBins. represents the lower limit of size for UnsortedBins. First, we list 10 critical expressions for as conditions for the piecewise function:

Then, we can build the mapping relations listed in Table 5. It should be noted that we only built the mapping relationship for the changed heap memory in the structural dimension rather than across the entire area. In order to construct a in a unified form, we count the primitives as part of solving the structural constraints. However, strictly speaking, these primitives will not directly bring changes in the heap memory structure, so we have not listed them in Table 5, but only built a numerical mapping in the related memory region for them (e.g., marking the target address of the writing primitive as controllable). In addition, we do not consider the influence of mmap [43] and reasonably assume that the allocated memory will not exceed the size of TopChunk (avoid triggering system calls to reduce speed).

For example, an allocator model gets input , and the current program state is , which satisfies . So according to the mapping relationship of , the model will output a new state , where the linked list changes to and returns an allocated chunk .

If entering a memory operation group , first of all, we will mutate their parameters. For a memory allocation primitive, we will mutate its size. For memory deallocation primitive, the mutation space of its parameter contains the addresses of all the allocated and deallocated chunks in the current state. For the writing primitive, the mutation space of its parameter contains the addresses to be controlled in the target state (the definition of target state will be given later), and the mutation space of width is the set [1, 2, 4, 8, 16, 32, 64, 128]. Then, we input these mutated MOPs into the heap allocator model one by one and process them according to the mapping relationship to obtain the final new program state.

With the help of the heap allocator model, after entering the MOG and the initial state of the program, we can infer the new state and match the new state with the target state (this matching method will be introduced later), and we can know whether the current MOG can make the memory state satisfy the structural constraint.

4.2.3. Solving the Migration Path

Our goal is to extract a MOG that satisfies the structural constraints from the program path to build . To achieve this goal, we should complete the following three processes: extract the MOG, solve , and consider the difficulty in driving . The method of extracting MOGs from the MOP graph was introduced earlier. Below, we first introduce the calculation method of the degree of state difference and explain how to find the sections that satisfy the structural constraints from MOGs by the degree of difference. Then, we introduce the concept of driving cost and explain how to use driving cost to reduce the difficulty of driving . Finally, we summarize the whole process for building.

(1) Degree of State Difference. We introduce the concept of state difference degree to measure the distance of memory structures between different states. Since there are different data structures in memory regions such as linked lists and the set of allocated chunks, we must define the distance of these typical data structures involved in detail.

The operation for the random access linked list is undirected, and chunks can be inserted and removed from any position on the linked list. Typical random access-linked lists include ptmalloc’s LargeBin linked list and UnsortedBin. For the two random access linked lists A and B, the degree of difference between the structures is defined in the following equation, where refers to the longest common subsequence distance (a special kind of the edit distance [44] which only considers insertion and deletion):

The direction of operation of the one-way access linked list is fixed, so insertion and deletion can only start from one end of the linked list. For example, insertion and deletion are performed in the same direction (FILO) for the FastBin list in ptmalloc and, however, in different directions (FIFO) for the SmallBin list. The definition of the distance between the two types of linked list structures can be expressed in the following equation, where and represent the length of the linked lists of and , respectively; for FILO-type linked lists, represents the longest matching length of after and are aligned backward in the direction of deletion. For FIFO-type linked lists, represents the length of the part that can completely match the subsequence of after the first same node of and from the insertion direction:

The degree of difference between unordered sets is defined in the following equation, where represents the number of elements which are located in but not in . For example, assuming that the target state requires 4 memory units to be controllable and only one of the memory units is currently controlled, the degree of difference between the current state and the target state is 3:

We total the degree of difference between the different data structures involved in the two states to obtain the total between the states.

From the structural constraints, we can acquire the initial state and corresponding target state (we can link them as a state migration demand expressed as ). By inputting the extracted MOG and the initial state into the heap allocator model, we can infer the new state and calculate the between the new state and the target state. If is 0, it indicates to the MOG that it satisfies the structural constraints. For example, there is a structural constraint for . First, we use the current program state as the initial state , and the chunk in the linked list is expressed as . Next, we ensure the target state . After receiving the operation group , from the heap allocator, the output state is , where the linked list changes to and returns the chunk . The between the new state and the target state is 0, indicating that we have found a .