We describe our process for automatic detection of performance changes for a software product in the presence of noise. A large collection of tests run periodically as changes to our software product are committed to our source repository, and we would like to identify the commits responsible for performance regressions. Previously, we relied on manual inspection of time series graphs to identify significant changes. That was later replaced with a threshold-based detection system, but neither system was sufficient for finding changes in performance in a timely manner. This work describes our recent implementation of a change point detection system built upon the E-Divisive means algorithm. The algorithm produces a list of change points representing significant changes from a given history of performance results. A human reviews the list of change points for actionable changes, which are then triaged for further inspection. Using change point detection has had a dramatic impact on our ability to detect performance changes. Quantitatively, it has dramatically dropped our false positive rate for performance changes, while qualitatively it has made the entire performance evaluation process easier, more productive (ex. catching smaller regressions), and more timely.

中文翻译：

---

软件性能测试中的变更点检测

我们描述了在存在噪声的情况下自动检测软件产品性能变化的过程。当我们的软件产品的更改提交到我们的源存储库时，会定期运行大量测试，我们希望确定导致性能回归的提交。以前，我们依靠手动检查时间序列图来识别重大变化。后来被基于阈值的检测系统所取代，但这两种系统都不足以及时发现性能变化。这项工作描述了我们最近实现的基于 E-Divisive 均值算法的变化点检测系统。该算法生成一个变化点列表，表示从给定的性能结果历史记录中的显着变化。人工审查可操作更改的更改点列表，然后对其进行分类以供进一步检查。使用变化点检测对我们检测性能变化的能力产生了巨大的影响。从数量上讲，它大大降低了我们对性能变化的误报率，而从质量上讲，它使整个性能评估过程变得更容易、更有效率（例如捕捉较小的回归），也更及时。