A systematic literature review (SLR) is an empirical method used to provide an overview of existing knowledge and to aggregate evidence within a domain. For computer science, several threats to the completeness of such reviews have been identified, leading to recommendations and guidelines on how to improve their quality. However, few studies address to what extent researchers can replicate an SLR. To conduct a replication, researchers have to first understand how the set of primary studies has been identified in the original study, and can ideally retrieve the same set when following the reported protocol. In this article, we focus on this initial step of a replication and report a two-fold empirical study: Initially, we performed a tertiary study using a sample of SLRs in computer science and identified what information that is needed to replicate the searches is reported. Based on the results, we conducted a descriptive, multi-case study on digital libraries to investigate to what extent these allow replications. The results reveal two threats to replications of SLRs: First, while researchers have improved the quality of their reports, relevant details are still missing—we refer to a reporting threat . Second, we found that some digital libraries are inconsistent in their query results—we refer to a searching threat . While researchers conducting a review can only overcome the first threat and the second may not be an issue for all kinds of replications, researchers should be aware of both threats when conducting, reviewing, and building on SLRs.

中文翻译：

---

搜索。审查。重复？对复制 SLR 搜索的威胁的实证研究

系统文献综述 (SLR) 是一种经验方法，用于提供对现有知识的概述并汇总某个领域内的证据。对于计算机科学，已经确定了对此类审查完整性的若干威胁，从而提出了有关如何提高其质量的建议和指南。然而，很少有研究涉及研究人员可以在多大程度上复制 SLR。要进行复制，研究人员必须首先了解原始研究中如何确定主要研究集，并且可以在遵循报告的协议时理想地检索相同的集。在本文中，我们专注于复制的初始步骤，并报告了两方面的实证研究：最初，我们使用计算机科学中的 SLR 样本进行了一项三级研究，并确定了报告复制搜索所需的信息。基于这些结果，我们对数字图书馆进行了描述性的多案例研究，以调查这些允许复制的程度。结果揭示了 SLR 复制面临的两个威胁：首先，虽然研究人员提高了报告的质量，但相关细节仍然缺失——我们指的是报告威胁。其次，我们发现一些数字图书馆的查询结果不一致——我们指的是搜索威胁。虽然进行审查的研究人员只能克服第一个威胁，而第二个威胁对于所有类型的复制可能都不是问题，但研究人员在进行、审查和构建 SLR 时应该意识到这两种威胁。