Crowdsourcing is an economic and efficient strategy aimed at collecting annotations of data through an online platform. Crowd workers with different expertise are paid for their service, and the task requester usually has a limited budget. How to collect reliable annotations for multilabel data and how to compute the consensus within budget are an interesting and challenging, but rarely studied, problem. In this article, we propose a novel approach to accomplish active multilabel crowd consensus (AMCC). AMCC accounts for the commonality and individuality of workers and assumes that workers can be organized into different groups. Each group includes a set of workers who share a similar annotation behavior and label correlations. To achieve an effective multilabel consensus, AMCC models workers' annotations via a linear combination of commonality and individuality and reduces the impact of unreliable workers by assigning smaller weights to their groups. To collect reliable annotations with reduced cost, AMCC introduces an active crowdsourcing learning strategy that selects sample-label-worker triplets. In a triplet, the selected sample and label are the most informative for the consensus model, and the selected worker can reliably annotate the sample at a low cost. Our experimental results on multilabel data sets demonstrate the advantages of AMCC over state-of-the-art solutions on computing crowd consensus and on reducing the budget by choosing cost-effective triplets.

中文翻译：

---

活跃的多标签人群共识。

众包是一种经济高效的策略，旨在通过在线平台收集数据注释。具有不同专业知识的人群工作人员为他们的服务付费，而任务请求者通常预算有限。如何为多标签数据收集可靠的注释以及如何在预算内计算共识是一个有趣且具有挑战性但很少研究的问题。在本文中，我们提出了一种实现主动多标签人群共识（AMCC）的新方法。AMCC 考虑了工人的共性和个性，并假设工人可以组织成不同的群体。每个组包括一组共享相似注释行为和标签相关性的工作人员。为了达成有效的多标签共识，AMCC 模拟了工人的 通过共性和个性的线性组合进行注释，并通过为他们的组分配较小的权重来减少不可靠工人的影响。为了以更低的成本收集可靠的注释，AMCC 引入了一种主动众包学习策略，该策略选择样本-标签-工人三元组。在三元组中，选定的样本和标签对共识模型的信息量最大，选定的工人可以以较低的成本可靠地对样本进行注释。我们在多标签数据集上的实验结果证明了 AMCC 在计算人群共识和通过选择具有成本效益的三元组来减少预算方面优于最先进的解决方案。AMCC 引入了一种主动的众包学习策略，可以选择样本-标签-工人三元组。在三元组中，选定的样本和标签对共识模型的信息量最大，选定的工人可以以较低的成本可靠地对样本进行注释。我们在多标签数据集上的实验结果证明了 AMCC 在计算人群共识和通过选择具有成本效益的三元组来减少预算方面优于最先进的解决方案。AMCC 引入了一种主动的众包学习策略，可以选择样本-标签-工人三元组。在三元组中，选定的样本和标签对共识模型的信息量最大，选定的工人可以以较低的成本可靠地对样本进行注释。我们在多标签数据集上的实验结果证明了 AMCC 在计算人群共识和通过选择具有成本效益的三元组来减少预算方面优于最先进的解决方案。