Programming errors that degrade the performance of systems are widespread, yet there is little tool support for analyzing these bugs. We present a method based on differential performance analysis---we find inputs for which the performance varies widely, despite having the same size. To ensure that the differences in the performance are robust (i.e. hold also for large inputs), we compare the performance of not only single inputs, but of classes of inputs, where each class has similar inputs parameterized by their size. Thus, each class is represented by a performance function from the input size to performance. Importantly, we also provide an explanation for why the performance differs in a form that can be readily used to fix a performance bug. The two main phases in our method are discovery with fuzzing and explanation with decision tree classifiers, each of which is supported by clustering. First, we propose an evolutionary fuzzing algorithm to generate inputs. For this fuzzing task, the unique challenge is that we not only need the input class with the worst performance, but rather a set of classes exhibiting differential performance. We use clustering to merge similar input classes which significantly improves the efficiency of our fuzzer. Second, we explain the differential performance in terms of program inputs and internals. We adapt discriminant learning approaches with clustering and decision trees to localize suspicious code regions. We applied our techniques to a set of applications. On a set of micro-benchmarks, we show that our approach outperforms state-of-the-art fuzzers in finding inputs to characterize the differential performance. On a set of case-studies, we discover and explain multiple performance bugs in popular machine learning frameworks. Four of these bugs, reported first in this paper, have since been fixed by the developers.

中文翻译：

---

检测和理解机器学习库中真实世界的差异性能错误

降低系统性能的编程错误很普遍，但很少有工具支持分析这些错误。我们提出了一种基于差异性能分析的方法——我们找到了性能差异很大的输入，尽管具有相同的大小。为了确保性能的差异是稳健的（即也适用于大输入），我们不仅比较了单个输入的性能，还比较了输入类的性能，其中每个类都有相似的输入，由它们的大小参数化。因此，每个类都由一个从输入大小到性能的性能函数表示。重要的是，我们还解释了为什么性能不同的形式可以很容易地用于修复性能错误。我们方法的两个主要阶段是使用模糊测试的发现和使用决策树分类器的解释，每个阶段都由聚类支持。首先，我们提出了一种进化模糊算法来生成输入。对于这个模糊测试任务，独特的挑战是我们不仅需要性能最差的输入类，还需要一组表现出不同性能的类。我们使用聚类来合并相似的输入类，这显着提高了我们的模糊器的效率。其次，我们根据程序输入和内部结构来解释不同的性能。我们采用聚类和决策树的判别学习方法来定位可疑代码区域。我们将我们的技术应用于一组应用程序。在一组微基准上，我们表明，我们的方法在寻找输入来表征差异性能方面优于最先进的模糊器。在一组案例研究中，我们发现并解释了流行机器学习框架中的多个性能错误。其中四个错误（在本文中首先报告）已被开发人员修复。