Comprehensive and fair performance evaluation of information retrieval systems represents an essential task for the current information age. Whereas Cranfield-based evaluations with benchmark datasets support development of retrieval models, significant evaluation efforts are required also for user-oriented systems that try to boost performance with an interactive search approach. This article presents findings from the 9th Video Browser Showdown, a competition that focuses on a legitimate comparison of interactive search systems designed for challenging known-item search tasks over a large video collection. During previous installments of the competition, the interactive nature of participating systems was a key feature to satisfy known-item search needs, and this article continues to support this hypothesis. Despite the fact that top-performing systems integrate the most recent deep learning models into their retrieval process, interactive searching remains a necessary component of successful strategies for known-item search tasks. Alongside the description of competition settings, evaluated tasks, participating teams, and overall results, this article presents a detailed analysis of query logs collected by the top three performing systems, SOMHunter, VIRET, and vitrivr. The analysis provides a quantitative insight to the observed performance of the systems and constitutes a new baseline methodology for future events. The results reveal that the top two systems mostly relied on temporal queries before a correct frame was identified. An interaction log analysis complements the result log findings and points to the importance of result set and video browsing approaches. Finally, various outlooks are discussed in order to improve the Video Browser Showdown challenge in the future.

中文翻译：

---

交互式搜索的统治是永恒的吗？2020 年视频浏览器对决的调查结果

信息检索系统的全面和公平的性能评估是当前信息时代的一项重要任务。尽管使用基准数据集的基于 Cranfield 的评估支持检索模型的开发，但对于试图通过交互式搜索方法提高性能的面向用户的系统，也需要进行大量的评估工作。本文介绍了第 9 届视频浏览器对决的结果，该竞赛侧重于对交互式搜索系统的合法比较，该系统旨在挑战大型视频集合中的已知项目搜索任务。在前几期比赛中，参与系统的交互性是满足已知项目搜索需求的关键特征，本文继续支持这一假设。尽管性能最佳的系统将最新的深度学习模型集成到其检索过程中，但交互式搜索仍然是已知项目搜索任务成功策略的必要组成部分。除了描述比赛设置、评估任务、参与团队和总体结果外，本文还详细分析了前三个执行系统 ​​SOMHunter、VIRET 和 vitrivr 收集的查询日志。该分析提供了对观察到的系统性能的定量洞察，并构成了未来事件的新基线方法。结果表明，在识别出正确的帧之前，前两个系统主要依赖于时间查询。交互日志分析补充了结果日志的发现，并指出了结果集和视频浏览方法的重要性。最后，讨论了各种前景，以改善未来的视频浏览器摊牌挑战。