Context: Systematic literature reviews (SLRs) rely on a rigorous and auditable methodology for minimizing biases and ensuring reliability. A common kind of bias arises when selecting studies using a set of inclusion/exclusion criteria. This bias can be decreased through dual revision, which makes the selection process more time-consuming and remains prone to generating bias depending on how each researcher interprets the inclusion/exclusion criteria. Objective: To reduce the bias and time spent in the study selection process, this paper presents a process for selecting studies based on the use of Cohen's Kappa statistic. We have defined an iterative process based on the use of this statistic during which the criteria are refined until obtain almost perfect agreement (k>0.8). At this point, the two researchers interpret the selection criteria in the same way, and thus, the bias is reduced. Starting from this agreement, dual review can be eliminated; consequently, the time spent is drastically shortened. Method: The feasibility of this iterative process for selecting studies is demonstrated through a tertiary study in the area of software engineering on works that were published from 2005 to 2018. Results: The time saved in the study selection process was 28% (for 152 studies) and if the number of studies is sufficiently large, the time saved tend asymptotically to 50%. Conclusions: Researchers and students may take advantage of this iterative process for selecting studies when conducting SLRs to reduce bias in the interpretation of inclusion and exclusion criteria. It is especially useful for research with few resources.

中文翻译：

---

软件工程中的系统文献综述 - 使用 Cohen 的 Kappa 统计增强研究选择过程

背景：系统性文献综述 (SLR) 依赖于严格且可审核的方法来最大程度地减少偏见并确保可靠性。使用一组纳入/排除标准选择研究时会出现一种常见的偏倚。这种偏见可以通过双重修订来减少，这使得选择过程更加耗时，并且仍然容易产生偏见，具体取决于每个研究人员如何解释纳入/排除标准。目标：为了减少研究选择过程中的偏差和花费的时间，本文提出了一种基于使用 Cohen's Kappa 统计量选择研究的过程。我们已经定义了一个基于使用该统计数据的迭代过程，在此过程中对标准进行细化，直到获得几乎完美的一致性 (k>0.8)。在此刻，两位研究人员以相同的方式解释选择标准，从而减少了偏差。从此协议开始，可以取消双重审查；因此，花费的时间大大缩短。方法：通过软件工程领域对 2005 年至 2018 年发表的作品的第三次研究，证明了这种选择研究的迭代过程的可行性。 结果：研究选择过程中节省的时间为 28%（对于 152 个研究) 并且如果研究数量足够大，则节省的时间逐渐趋于 50%。结论：研究人员和学生在进行 SLR 时可以利用这种迭代过程来选择研究，以减少对纳入和排除标准的解释偏差。它对于资源很少的研究特别有用。