Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of crowdsourcing (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if any worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.

中文翻译：

---

探索用于系统评论的众包引证筛选。

系统评价越来越多地用于为医疗保健决策提供信息，但生产成本很高。我们探索使用众包（通过网络将任务分配给未经培训的工人）以减少筛查引文的成本。我们使用Amazon Mechanical Turk作为平台，并以4个先前进行的系统评估为例。对于每次引用，工作人员都会回答4或5个与资格标准相同的问题。我们使用9个算法中的1个将来自多个工作人员的响应汇总到一个包括或排除引用的总体决策中，并将这些算法的性能与经过培训的专家的相应决策进行比较。最具包容性的算法（指定引用的相关的，如果任何工作者确实）确定了最终纳入评论的95％至99％的引用，但排除了68％至82％的无关引用。其他算法增加了以相关成本将无关文章排除在外的比例，从而增加了相关研究的纳入。人群工作者在4到17天内完成了筛查，费用为460到2220美元，与经过培训的专家相比，费用降低了88％。众包可能是降低识别文献以进行系统评价的成本的有用方法。