From the early developments of machines for reasoning and decision making in higher-level information fusion, there was a need for a systematic and reliable evaluation of their performance. Performance evaluation is important for comparison and assessment of alternative solutions to real-world problems. In this paper we focus on one aspect of performance assessment for reasoning under uncertainty: the accuracy of the resulting belief (prediction or estimate). We propose a framework for assessment based on the assumption that the system under investigation is uncertain only due to stochastic variability (randomness), which is partially known. In this context we formulate a distance measure between the “ground truth” and the output of an automated system for reasoning in the framework of one of the non-additive uncertainty formalisms (such as imprecise probability theory, belief function theory or possibility theory). The proposed assessment framework is demonstrated with a simple numerical example.

从高级信息融合中推理和决策机器的早期发展开始，就需要对其性能进行系统和可靠的评估。绩效评估对于比较和评估实际问题的替代解决方案很重要。在本文中，我们专注于在不确定性下进行推理的绩效评估的一个方面：结果置信度（预测或估计）的准确性。我们提出了一个评估框架，该假设基于以下假设：所研究的系统仅由于随机可变性（随机性）才是不确定的，而这种随机性是部分已知的。在这种情况下，我们在一种非累加的不确定性形式主义（例如不精确概率论，信念函数论或可能性论）的框架内，制定了“基本事实”与自动化系统输出之间的距离度量，以进行推理。所提出的评估框架通过一个简单的数字示例进行了演示。