Assessors make preference judgments faster and more consistently than graded judgments. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments. Unfortunately, preference judgments can require more than linear effort to fully order a pool of items, and evaluation measures for preference judgments are not as well established as those for graded judgments, such as NDCG. In this article, we explore the assessment process for partial preference judgments, with the aim of identifying and ordering the top items in the pool, rather than fully ordering the entire pool. To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure. We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named compatibility . This new measure has its most striking impact when comparing modern neural rankers, where it is able to recognize significant improvements in quality that would otherwise be missed by NDCG.

中文翻译：

---

评估首要偏好

与分级判断相比，评估员做出偏好判断更快、更一致。偏好判断还可以识别在分级判断下看起来等同的项目之间的区别。不幸的是，偏好判断可能需要更多的线性努力来完全排序一组项目，并且偏好判断的评估措施不如分级判断的评估措施，如 NDCG。在本文中，我们探讨了部分偏好判断的评估过程，目的是识别和排序池中的顶部项目，而不是对整个池进行完全排序。为了衡量排名器的性能，我们通过应用排名相似性度量将其输出与此首选排序进行比较。兼容性. 在比较现代神经排序器时，这项新措施具有最显着的影响，它能够识别出质量的显着改进，否则 NDCG 会错过这些改进。