Use of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 years since that work started, use of test collections is a de facto standard of evaluation. This monograph surveys the research conducted and explains the methods and measures devised for evaluation of retrieval systems, including a detailed look at the use of statistical significance testing in retrieval experimentation. This monograph reviews more recent examinations of the validity of the test collection approach and evaluation measures as well as outlining trends in current research exploiting query logs and live labs. At its core, the modern-day test collection is little different from the structures that the pioneering researchers in the 1950s and 1960s conceived of. This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.

使用测试集和评估措施来评估信息检索系统的有效性可追溯到1950年代初。在这项工作开始以来的近60年中，使用测试集实际上是评估的标准。这本专论对进行的研究进行了调查，并解释了为评估检索系统而设计的方法和措施，包括详细介绍了统计意义检验在检索实验中的使用。这本专论回顾了有关测试收集方法和评估措施的有效性的最新检查，并概述了利用查询日志和实时实验室进行当前研究的趋势。本质上，现代的测试集与1950年代和1960年代的开创性研究人员所设想的结构几乎没有什么不同。本教程和评论表明，尽管年代久远，但这种长期的评估方法仍然是用于检索研究的非常有价值的工具。