ABSTRACT In this article, we report on the lack of reliability of explicit user feedback and its interpretation in the light of system evaluation. It is known, that given feedback strongly depends on the situational context. But also when many contextual factors are held constant, user feedback still proves to be unreliable. This impacts the evaluation of predictive algorithms since it is not clear whether a deviation between a user response and its corresponding prediction can be seen as a flaw by the system or just as usual ‘human uncertainty’. As a result, the perspective on the evaluation of adaptive systems basically changes. The main goal of this article is to demonstrate that simply increasing the amount of explicit feedback is not the key to sustainable system design innovation, as long as that information is not appropriately evaluated. To this end, we will exploit a novel probabilistic approach of processing user feedback and identify biasing effects on accuracy metrics, error probabilities for system rankings as well as natural limitations of evaluation. Finally, we will discuss possible solution strategies and give advice for handling explicit user feedback that is associated with uncertainty.

中文翻译：

---

显式用户反馈中的人为不确定性及其对准确预测和个性化比较评估的影响

摘要在本文中，我们报告了明确的用户反馈缺乏可靠性及其根据系统评估的解释。众所周知，给定的反馈很大程度上取决于情境上下文。但是，当许多上下文因素保持不变时，用户反馈仍然被证明是不可靠的。这会影响预测算法的评估，因为尚不清楚用户响应与其相应预测之间的偏差是否可以被系统视为缺陷，或者就像通常的“人类不确定性”。因此，对适应性系统评价的观点发生了根本性的变化。本文的主要目标是证明，只要信息没有得到适当评估，简单地增加显式反馈的数量并不是可持续系统设计创新的关键。为此，我们将利用一种新的概率方法来处理用户反馈，并识别对准确度指标、系统排名的错误概率以及评估的自然限制的偏差影响。最后，我们将讨论可能的解决策略并给出处理与不确定性相关的明确用户反馈的建议。