Benchmarks are an integral part of machine learning development. However, the most common benchmarks share several limitations. For example, the difference in performance between two models has no probabilistic interpretation, it makes no sense to compare such differences between data sets and there is no reference point that indicates a significant performance improvement. Here we introduce an Elo-based predictive power meta-score that is built on other performance measures and allows for interpretable comparisons of models. Differences between this score have a probabilistic interpretation and can be compared directly between data sets. Furthermore, this meta-score allows for an assessment of ranking fitness. We prove the properties of the Elo-based predictive power meta-score and support them with empirical results on a large-scale benchmark of 30 classification data sets. Additionally, we propose a unified benchmark ontology that provides a uniform description of benchmarks.

基准是机器学习开发的一个组成部分。然而，最常见的基准有几个限制。例如，两个模型之间的性能差异没有概率解释，比较数据集之间的这种差异是没有意义的，并且没有表明性能显着提高的参考点。在这里，我们介绍了一个基于 Elo 的预测能力元分数，它建立在其他性能指标之上，并允许对模型进行可解释的比较。该分数之间的差异具有概率解释，可以直接在数据集之间进行比较。此外，这个元分数允许评估排名适应度。我们证明了基于 Elo 的预测能力元分数的属性，并在 30 个分类数据集的大规模基准上用经验结果支持它们。此外，我们提出了一个统一的基准本体，它提供了基准的统一描述。