Abstract

Nondeterministic measurements are common in real-world scenarios: the performance of a stochastic optimization algorithm or the total reward of a reinforcement learning agent in a chaotic environment are just two examples in which unpredictable outcomes are common. These measures can be modeled as random variables and compared among each other via their expected values or more sophisticated tools such as null hypothesis statistical tests. In this article, we propose an alternative framework to visually compare two samples according to their estimated cumulative distribution functions. First, we introduce a dominance measure for two random variables that quantifies the proportion in which the cumulative distribution function of one of the random variables stochastically dominates the other one. Then, we present a graphical method that decomposes in quantiles (i) the proposed dominance measure and (ii) the probability that one of the random variables takes lower values than the other. With illustrative purposes, we reevaluate the experimentation of an already published work with the proposed methodology and we show that additional conclusions—missed by the rest of the methods—can be inferred. Additionally, the software package RVCompare was created as a convenient way of applying and experimenting with the proposed framework.

摘要

不确定性测量在现实场景中很常见：随机优化算法的性能或混沌环境中强化学习代理的总奖励只是不可预测结果常见的两个例子。这些度量可以建模为随机变量，并通过其预期值或更复杂的工具（例如零假设统计检验）相互比较。在本文中，我们提出了一个替代框架，根据估计的累积分布函数直观地比较两个样本。首先，我们引入两个随机变量的优势测度，该测度量化其中一个随机变量的累积分布函数随机支配另一个随机变量的比例。然后，我们提出了一种图形方法，可以将（i）所提出的优势度量和（ii）其中一个随机变量的值低于另一个的概率分解为分位数。出于说明的目的，我们使用所提出的方法重新评估了已发表的作品的实验，并且我们表明可以推断出其他方法遗漏的其他结论。另外，该软件包RVCompare 的创建是为了应用和试验所提出的框架的便捷方式。