Context

Software performance may suffer regressions caused by source code changes. Measuring performance at each new software version is useful for early detection of performance regressions. However, systematically running benchmarks is often impractical (e.g., long running execution, prioritizing functional correctness over non-functional).

Objective

In this article, we propose Horizontal Profiling, a sampling technique to predict when a new revision may cause a regression by analyzing the source code and using run-time information of a previous version. The goal of Horizontal Profiling is to reduce the performance testing overhead by benchmarking just software versions that contain costly source code changes.

Method

We present an evaluation in which we apply Horizontal Profiling to identify performance regressions of 17 software projects written in the Pharo programming language, totaling 1,288 software versions.

Results

Horizontal Profiling detects more than 80% of the regressions by benchmarking less than 20% of the versions. In addition, our experiments show that Horizontal Profiling has better precision and executes the benchmarks in less versions that the state of the art tools, under our benchmarks.

Conclusions

We conclude that by adequately characterizing the run-time information of a previous version, it is possible to determine if a new version is likely to introduce a performance regression or not. As a consequence, a significant fraction of the performance regressions are identified by benchmarking only a small fraction of the software versions.

中文翻译：

---

为性能回归测试确定版本优先级：Pharo案例

语境

软件性能可能会因源代码更改而遭受退步。测量每个新软件版本的性能对于及早发现性能下降很有用。但是，系统地运行基准测试通常是不切实际的（例如，长时间运行的执行，将功能正确性置于非功能性之上）。

目的

在本文中，我们提出了“水平剖析”，这是一种采样技术，可以通过分析源代码并使用先前版本的运行时信息来预测新修订何时可能导致回归。水平剖析的目的是通过仅对包含昂贵源代码更改的软件版本进行基准测试来减少性能测试的开销。

方法

我们提供了一种评估方法，其中我们应用水平剖析来识别以Pharo编程语言编写的17个软件项目的性能回归，总共有1,288个软件版本。

结果

通过对少于20％的版本进行基准测试，水平剖析可检测80％以上的回归。此外，我们的实验表明，与我们的基准测试相比，水平剖析具有更高的精度，并且可以以更少的版本执行基准测试。

结论

我们得出的结论是，通过充分表征先前版本的运行时信息，可以确定新版本是否可能导致性能下降。结果，仅通过对软件版本的一小部分进行基准测试，就可以确定很大一部分性能下降。