Abstract

Data competitions have become a popular and cost-effective approach for crowdsourcing versatile solutions from diverse expertise. Current practice relies on the simple leaderboard scoring based on a given set of competition data for ranking competitors and distributing the prize. However, a disadvantage of this practice in many competitions is that a slight difference in the scores due to the natural variability of the observed data could result in a much larger difference in the prize amounts. In this article, we propose a new strategy to quantify the uncertainty in the rankings and scores from using different data sets that share common characteristics with the provided competition data. By using a bootstrap approach to generate many comparable data sets, the new method has four advantages over current practice. During the competition, it provides a mechanism for competitors to get feedback about the uncertainty in their relative ranking. After the competition, it allows the host to gain a deeper understanding of the algorithm performance and their robustness across representative data sets. It also offers a transparent mechanism for prize distribution to reward the competitors more fairly with superior and robust performance. Finally, it has the additional advantage of being able to explore what results might have looked like if competition goals evolved from their original choices. The implementation of the strategy is illustrated with a real data competition hosted by Topcoder on urban radiation search.

摘要

数据竞赛已成为一种流行且具有成本效益的方法，用于从不同专业知识中众包多功能解决方案。当前的做法依赖于基于给定比赛数据集的简单排行榜评分，用于对竞争对手进行排名并分配奖品。然而，这种做法在许多比赛中的一个缺点是，由于观察到的数据的自然可变性，分数的微小差异可能会导致奖金金额的更大差异。在本文中，我们提出了一种新策略，通过使用与提供的比赛数据具有共同特征的不同数据集来量化排名和分数的不确定性。通过使用 bootstrap 方法生成许多可比较的数据集，新方法与当前实践相比具有四个优点。比赛期间，它为竞争者提供了一种机制，以获取有关其相对排名不确定性的反馈。比赛结束后，它允许主持人更深入地了解算法性能及其跨代表性数据集的鲁棒性。它还提供了一种透明的奖品分配机制，以卓越和稳健的表现更公平地奖励竞争对手。最后，它还有一个额外的优势，那就是能够探索如果比赛目标从最初的选择演变而来，结果可能会是什么样子。该策略的实施通过由 Topcoder 主办的关于城市辐射搜索的真实数据竞赛来说明。它允许主机更深入地了解算法性能及其跨代表性数据集的稳健性。它还提供了一种透明的奖品分配机制，以卓越和稳健的表现更公平地奖励竞争对手。最后，它还有一个额外的优势，那就是能够探索如果比赛目标从最初的选择演变而来，结果可能会是什么样子。该策略的实施通过由 Topcoder 主办的关于城市辐射搜索的真实数据竞赛来说明。它允许主机更深入地了解算法性能及其跨代表性数据集的稳健性。它还提供了一种透明的奖品分配机制，以卓越和稳健的表现更公平地奖励竞争对手。最后，它还有一个额外的优势，那就是能够探索如果比赛目标从最初的选择演变而来，结果可能会是什么样子。该策略的实施通过由 Topcoder 主办的关于城市辐射搜索的真实数据竞赛来说明。它还有一个额外的优势，那就是能够探索如果比赛目标从最初的选择演变而来，结果可能会是什么样子。该策略的实施通过由 Topcoder 主办的关于城市辐射搜索的真实数据竞赛来说明。它还有一个额外的优势，那就是能够探索如果比赛目标从最初的选择演变而来，结果可能会是什么样子。该策略的实施通过由 Topcoder 主办的关于城市辐射搜索的真实数据竞赛来说明。