In the setting where a group of agents is asked a single subjective multi-choice question (e.g. which one do you prefer? cat or dog?), we are interested in evaluating the quality of the collected feedback. However, the collected statistics are not sufficient to reflect how informative the feedback is since fully informative feedback (equal affection of the choices) and fully uninformative feedback (random selection) have the same uniform statistics. Here we distinguish the above two scenarios by additionally asking for respondents' predictions about others' choices. We assume that informative respondents' predictions strongly depend on their own choices while uninformative respondents' do not. With this assumption, we propose a new definition for uninformative feedback and correspondingly design a family of evaluation metrics, called f-variety, for group-level feedback which can 1) distinguish informative feedback and uninformative feedback (separation) even if their statistics are both uniform and 2) decrease as the ratio of uninformative respondents increases (monotonicity). We validate our approach both theoretically and numerically. Moreover, we conduct two real-world case studies about 1) comparisons about athletes and 2) comparisons about stand-up comedians to show the superiority of our approach.

中文翻译：

---

平等感情或随机选择：从群体角度看主观反馈的质量

在向一组座席提出单个主观选择题的情况下（例如，您更喜欢哪个？猫还是狗？），我们对评估收集到的反馈的质量很感兴趣。但是，收集的统计信息不足以反映反馈的信息量，因为完全信息反馈（选择的相等影响）和完全非信息反馈（随机选择）具有相同的统一统计信息。在这里，我们通过另外询问受访者对他人选择的预测来区分以上两种情况。我们假设信息丰富的受访者的预测很大程度上取决于他们自己的选择，而信息不丰富的受访者则没有。以此假设为基础，我们为无信息的反馈提出了新的定义，并相应地设计了一系列评估指标，称为f变量，用于组级别的反馈，它可以1）区分信息反馈和非信息反馈（分离），即使它们的统计数据均是统一的； 2）随着非信息受访者比例的增加而降低（单调）。我们在理论上和数值上都验证了我们的方法。此外，我们进行了两个关于1）关于运动员的比较和2）关于站立喜剧演员的比较的现实案例研究，以显示我们方法的优越性。