Performance evaluation is a key task in computing and communication systems. Benchmarking is one of the most common techniques for evaluation purposes, where the performance of a set of representative applications is used to infer system responsiveness in a general usage scenario. Unfortunately, most benchmarking suites are limited to a reduced number of applications, and in some cases, rigid execution configurations. This makes it hard to extrapolate performance metrics for a general-purpose architecture, supposed to have a multi-year lifecycle, running dissimilar applications concurrently. The main culprit of this situation is that current benchmark-derived metrics lack generality, statistical soundness and fail to represent general-purpose environments. Previous attempts to overcome these limitations through random app mixes significantly increase computational cost (workload population shoots up), making the evaluation process barely affordable. To circumvent this problem, in this article we present a more elaborate performance evaluation methodology named BenchCast. Our proposal provides more representative performance metrics, but with a drastic reduction of computational cost, limiting app execution to a small and representative fraction marked through code annotation. Thanks to this labeling and making use of synchronization techniques, we generate heterogeneous workloads where every app runs simultaneously inside its Region Of Interest, making a few execution seconds highly representative of full application execution.

中文翻译：

---

通过异构、基于样本的基准测试快速、准确地评估处理器


性能评估是计算和通信系统中的一项关键任务。基准测试是用于评估目的的最常用技术之一，其中一组代表性应用程序的性能用于推断一般使用场景中的系统响应能力。不幸的是，大多数基准测试套件仅限于减少数量的应用程序，并且在某些情况下仅限于严格的执行配置。这使得很难推断通用架构的性能指标，该架构应该具有多年的生命周期，同时运行不同的应用程序。造成这种情况的主要原因是当前的基准衍生指标缺乏通用性、统计可靠性，并且无法代表通用环境。之前通过随机应用程序组合来克服这些限制的尝试显着增加了计算成本（工作负载数量激增），使得评估过程几乎无法承受。为了解决这个问题，在本文中我们提出了一种更精细的性能评估方法，名为 BenchCast。我们的提案提供了更具代表性的性能指标，但大大降低了计算成本，将应用程序执行限制为通过代码注释标记的较小且具有代表性的部分。由于这种标记和同步技术的使用，我们生成了异构工作负载，其中每个应用程序在其感兴趣的区域内同时运行，使得几秒钟的执行高度代表了完整的应用程序执行。