Online crowdsourcing provides a scalable and inexpensive means to collect knowledge (e.g. labels) about various types of data items (e.g. text, audio, video). However, it is also known to result in large variance in the quality of recorded responses which often cannot be directly used for training machine learning systems. To resolve this issue, a lot of work has been conducted to control the response quality such that low-quality responses cannot adversely affect the performance of the machine learning systems. Such work is referred to as the quality control for crowdsourcing. Past quality control research can be divided into two major branches: quality control mechanism design and statistical models. The first branch focuses on designing measures, thresholds, interfaces and workflows for payment, gamification, question assignment and other mechanisms that influence workers' behaviour. The second branch focuses on developing statistical models to perform effective aggregation of responses to infer correct responses. The two branches are connected as statistical models (i) provide parameter estimates to support the measure and threshold calculation, and (ii) encode modelling assumptions used to derive (theoretical) performance guarantees for the mechanisms. There are surveys regarding each branch but they lack technical details about the other branch. Our survey is the first to bridge the two branches by providing technical details on how they work together under frameworks that systematically unify crowdsourcing aspects modelled by both of them to determine the response quality. We are also the first to provide taxonomies of quality control papers based on the proposed frameworks. Finally, we specify the current limitations and the corresponding future directions for the quality control research.

中文翻译：

---

众包质量控制统计建模与设计方法技术综述

在线众包提供了一种可扩展且廉价的方式来收集有关各种类型数据项（例如文本、音频、视频）的知识（例如标签）。然而，众所周知，记录响应的质量会产生很大的差异，这通常不能直接用于训练机器学习系统。为了解决这个问题，已经进行了大量工作来控制响应质量，使得低质量的响应不会对机器学习系统的性能产生不利影响。这种工作被称为众包的质量控制。过去的质量控制研究可以分为两大分支：质量控制机制设计和统计模型。第一个分支专注于设计支付、游戏化、问题分配和其他影响工人行为的机制。第二个分支侧重于开发统计模型以执行有效的响应聚合以推断正确的响应。这两个分支连接为统计模型 (i) 提供参数估计以支持测量和阈值计算，以及 (ii) 编码建模假设，用于推导出机制的（理论）性能保证。有关于每个分支的调查，但缺乏关于另一个分支的技术细节。我们的调查是第一个通过提供有关它们如何在框架下协同工作的技术细节来弥合这两个分支的调查，这些框架系统地统一了由它们两个建模的众包方面以确定响应质量。我们也是第一个根据提议的框架提供质量控制论文分类的人。最后，我们指定了质量控制研究的当前限制和相应的未来方向。