Background

In the screening phase of systematic review, researchers use detailed inclusion/exclusion criteria to decide whether each article in a set of candidate articles is relevant to the research question under consideration. A typical review may require screening thousands or tens of thousands of articles in and can utilize hundreds of person-hours of labor.

Methods

Here we introduce SWIFT-Active Screener, a web-based, collaborative systematic review software application, designed to reduce the overall screening burden required during this resource-intensive phase of the review process. To prioritize articles for review, SWIFT-Active Screener uses active learning, a type of machine learning that incorporates user feedback during screening. Meanwhile, a negative binomial model is employed to estimate the number of relevant articles remaining in the unscreened document list. Using a simulation involving 26 diverse systematic review datasets that were previously screened by reviewers, we evaluated both the document prioritization and recall estimation methods.

Results

On average, 95% of the relevant articles were identified after screening only 40% of the total reference list. In the 5 document sets with 5,000 or more references, 95% recall was achieved after screening only 34% of the available references, on average. Furthermore, the recall estimator we have proposed provides a useful, conservative estimate of the percentage of relevant documents identified during the screening process.

Conclusion

SWIFT-Active Screener can result in significant time savings compared to traditional screening and the savings are increased for larger project sizes. Moreover, the integration of explicit recall estimation during screening solves an important challenge faced by all machine learning systems for document screening: when to stop screening a prioritized reference list. The software is currently available in the form of a multi-user, collaborative, online web application.

中文翻译：

---

SWIFT-Active Screener：通过主动学习和集成召回评估加速文档筛选

 背景

在系统评价的筛选阶段，研究人员使用详细的纳入/排除标准来确定一组候选文章中的每篇文章是否与正在考虑的研究问题相关。一次典型的审查可能需要筛选数千或数万篇文章，并且可能需要花费数百个工时。

 方法

在此，我们介绍 SWIFT-Active Screener，这是一款基于网络的协作系统审核软件应用程序，旨在减轻审核过程中资源密集型阶段所需的总体筛选负担。为了优先审查文章，SWIFT-Active Screener 使用主动学习，这是一种机器学习，在筛选过程中纳入了用户反馈。同时，采用负二项式模型来估计未筛选文档列表中剩余的相关文章的数量。我们使用包含审阅者之前筛选的 26 个不同系统审阅数据集的模拟，评估了文档优先级排序和召回估计方法。

 结果

平均而言，仅筛选了总参考文献列表的 40% 后，就识别出了 95% 的相关文章。在包含 5,000 条或更多参考文献的 5 个文档集中，平均仅筛选 34% 的可用参考文献即可实现 95% 的召回率。此外，我们提出的召回估计器对筛选过程中识别的相关文档的百分比提供了有用的、保守的估计。

 结论

与传统筛查相比，SWIFT-Active Screener 可以显着节省时间，并且项目规模越大，节省的时间就越多。此外，在筛选过程中集成显式召回估计解决了所有用于文档筛选的机器学习系统面临的一个重要挑战：何时停止筛选优先参考列表。该软件目前以多用户协作在线网络应用程序的形式提供。