This article conducts user evaluation to study the performance difference between interactive and automatic search. Particularly, the study aims to provide empirical insights of how the performance landscape of video search changes, with tens of thousands of concept detectors freely available to exploit for query formulation. We compare three types of search modes: free-to-play (i.e., search from scratch), non-free-to-play (i.e., search by inspecting results provided by automatic search), and automatic search including concept-free and concept-based retrieval paradigms. The study involves a total of 40 participants; each performs interactive search over 15 queries of various difficulty levels using two search modes on the IACC.3 dataset provided by TRECVid organizers. The study suggests that the performance of automatic search is still far behind interactive search. Furthermore, providing users with the result of automatic search for exploration does not show obvious advantage over asking users to search from scratch. The study also analyzes user behavior to reveal insights of how users compose queries, browse results, and discover new query terms for search, which can serve as guideline for future research of both interactive and automatic search.

中文翻译：

---

交互式搜索与自动搜索

本文通过用户评估来研究交互式搜索和自动搜索之间的性能差异。特别是，该研究旨在提供有关视频搜索性能如何变化的经验见解，数以万计的概念检测器可免费用于查询制定。我们比较了三种搜索模式：free-to-play（即从头开始搜索）、non-free-to-play（即通过检查自动搜索提供的结果进行搜索）和自动搜索（包括无概念和概念）基于检索范式。该研究共有 40 名参与者；每个人都使用 TRECVid 组织者提供的 IACC.3 数据集上的两种搜索模式对 15 个不同难度级别的查询进行交互式搜索。该研究表明，自动搜索的性能仍然远远落后于交互式搜索。此外，与要求用户从头开始搜索相比，为用户提供自动搜索探索的结果并没有显示出明显的优势。该研究还分析了用户行为，以揭示用户如何撰写查询、浏览结果和发现新的搜索查询词，这可以作为未来交互式和自动搜索研究的指南。