PARIS: Predicting application resilience using machine learning

Highlights

• PARIS presents a machine-learning method to predict HPC application resilience.

• PARIS captures the relationship between application characteristics and resilience.

• PARIS avoids random fault injection and provides high prediction accuracy.

• PARIS provides a foundation for resilience modeling using machine learning.

Abstract

The traditional method to study application resilience to errors in HPC applications uses fault injection (FI), a time-consuming approach. While analytical models have been built to overcome the inefficiencies of FI, they lack accuracy. In this paper, we present PARIS, a machine-learning method to predict application resilience that avoids the time-consuming process of random FI and provides higher prediction accuracy than analytical models. PARIS captures the implicit relationship between application characteristics and application resilience, which is difficult to capture using most analytical models. We overcome many technical challenges for feature construction, extraction, and selection to use machine learning in our prediction approach. Our evaluation on 16 HPC benchmarks shows that PARIS achieves high prediction accuracy. PARIS is up to 450x faster than random FI (49x on average). Compared to the state-of-the-art analytical model, PARIS is at least 63% better in terms of accuracy and has comparable execution time on average.

Introduction

As high performance computing (HPC) systems increase in scale, they become more susceptible to transient faults [6] due to feature size shrinking, lower voltages, and increasing densities in hardware infrastructures [59]. As a result, scientific applications running at extreme scales apply different resilience methods to tolerate frequent soft errors. Applying these methods to a given application often requires a deep understanding of the resilience of the application.

The common practice to study application resilience to errors in HPC systems is Fault Injection (FI) [11], [17], [62], [63], [64]. This approach uses a large amount of random injections, each of which randomly selects an instruction, and then triggers bit flips at the instruction input or output operands during application execution. Statistical results are then used to quantify application resilience.

While FI works in practice and is widely used in resilience studies, a key problem of this approach is that it is highly time consuming. To illustrate the problem, consider an application that runs for 6 hours—a common execution time for a large-scale scientific simulation [26]. Using statistical analysis (e.g., using [45]), the number of random FIs to obtain a low margin of error (e.g., 1%–3%) is in the order of thousands of injections. Thus, the total FI campaign could last several days. For multi-threaded or multi-process applications, this time is significantly higher since random faults must be injected into different threads or processes.

To address the limitations of FI, researchers have built error-propagation analytical models [46], which are faster than FI in estimating application resilience. However, they lack of accuracy as they estimate application resilience to errors based on analysis of possible errors in individual instructions. The analysis inaccuracy at individual instructions is accumulated, causing low accuracy to estimate the whole application resilience. Furthermore, these models do not consider effects of resilience computation patterns (e.g., dead corrupted locations and repeated addition [32]). Studying those patterns demands analyzing multiple instructions together, while most existing analytical models analyze instructions in isolation.

In summary, the community lacks a fundamental approach that enables fast and accurate evaluation of application resilience. In this paper, we present a novel framework called PARIS,¹ which avoids the time-consuming process of randomly selecting and executing many injections (as in FI), and provides higher prediction accuracy than analytical models, making it a unique solution to the problem. In essence, PARIS uses a machine learning model to predict application resilience, which provides several advantages. First, machine learning models, once trained, can be repeatedly used for any fault manifestations – silent data corruption (SDC), interruptions, and success cases – for new, previously unseen applications. Therefore, PARIS avoids a large amount of repeated fault injection tests, which leads to high efficiency in comparison to FI. Second, machine learning models can capture the implicit relationship between application characteristics (e.g., intensity of resilience computation patterns) and application resilience, which is difficult to capture by analytical models.

The most challenging part of using the machine learning approach is to build effective features. We use the following methods to construct a feature vector of 30 features.

First, we count the number of instruction instances within each instruction type as a feature; instruction instances are dynamic execution of instructions. We characterize instructions in such a way because different instruction types show different resilience to errors [12], [37]. To reduce the number of features, we classify instruction types into four representative and discriminative groups in terms of the functionality of instructions. This reduction of features reduces the training complexity and avoids undertraining.

Second, we count resilience computation patterns as features. Guo et al. [32] discover six resilience computation patterns from HPC applications. Those patterns are considered the fundamental reason for application resilience. Four of those patterns are based on individual instructions, and can be included as features using the above instruction type-based approach. The remaining two (“dead locations” and “repeated addition”) contain more than one instruction and cannot be captured by examining instructions individually. To efficiently count the two patterns, we introduce optimization techniques to avoid repeatedly scanning the instruction trace and find correlation between instructions.

Third, we introduce instruction execution order information into features to improve modeling accuracy. Execution order information is important to application resilience, because error propagation is highly correlated to the order and type of operations. Inspired by “N-gram” technique [16], [56] in computational linguistics, we embedded the sequence of instruction chunks into features to introduce execution order of instructions. Our evaluation shows that having execution order information decreases prediction error by up to 30%.

Fourth, we introduce resilience weight when counting instruction instances. Different instruction instances, even though they have the same instruction type, can have different capabilities to tolerate faults. Resilience weight quantifies the resilience difference of those instruction instances. Introducing resilience weight decreases prediction error by 13% on average when predicting the rate of some fault manifestation (particularly, the interruption rate).

Based upon the above features, we use feature selection techniques to sort and further reduce features. We perform ablation study to understand the sensitivity of features to prediction accuracy. We reveal significance of memory-related instructions and data overwriting to application resilience.

In summary, our contributions are as follows.

• We present PARIS, a machine learning-based approach to predict application resilience. Our method breaks the fundamental tradeoff between evaluation speed and accuracy in the existing common practice to estimate application resilience.

• We develop a framework and overcome a series of technical challenges for feature construction, extraction and selection. We reveal how to use machine learning to effectively and efficiently model application resilience.

• We test our model on 16 benchmarks. We find that our approach is up to 450x faster than random FI (49x on average). The model has high prediction accuracy: a prediction error of 8.5% and 22% on average for predicting success rate and interruption rate (excluding two obvious outliers) respectively. We compare PARIS with Trident [46] (the state-of-the-art analytical model): PARIS can predict any fault manifestation rate (SDC, interruptions, and success), while Trident only predicts SDC rate; PARIS is at least 63% better than Trident in terms of accuracy for predicting SDC rate, and has comparable execution time (but faster for 12 out of the 16 benchmarks with 15x speedup on average).