Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

中文翻译：

---

哪些学术检索系统适合系统评价或荟萃分析？评估 Google Scholar、PubMed 和 26 个其他资源的检索质量。

严格的证据识别对于系统评价和荟萃分析（证据合成）至关重要，因为相关研究的样本选择决定了评价的结果、有效性和解释力。然而，允许访问这些证据的搜索系统提供了不同程度的精确度、召回率和再现性，并且也需要不同程度的努力。迄今为止，尚不清楚哪些搜索系统最适合证据合成以及原因。关于选择哪些搜索引擎和书目数据库进行系统检索的建议很有限，并且缺乏系统的、实证的性能评估。本研究调查并比较了 28 个广泛使用的学术搜索系统（包括 Google Scholar、PubMed 和 Web of Science）的系统搜索质量。一种基于查询的新颖方法测试用户与每个系统交互和检索记录的能力。该研究首次展示了搜索系统在精确度、召回率和再现性方面能够有效且高效地执行（布尔）搜索的程度。我们发现搜索系统的性能存在显着差异，这意味着它们在系统搜索中的可用性各不相同。事实上，在不增加大量警告的情况下，只有一半的搜索系统被分析，并且只有少数开放存取数据库可以被推荐用于证据综合。特别是，我们的研究结果证明了为什么谷歌学术不适合作为主要搜索系统。我们呼吁数据库所有者认识到证据合成的要求，并呼吁学术期刊重新评估系统评价的质量要求。我们的研究结果旨在支持研究人员更好地进行搜索以更好地综合证据。