With the rapidly increasing interest in machine learning based solutions for automatic image annotation, the availability of reference annotations for algorithm training is one of the major bottlenecks in the field. Crowdsourcing has evolved as a valuable option for low-cost and large-scale data annotation; however, quality control remains a major issue which needs to be addressed. To our knowledge, we are the first to analyze the annotation process to improve crowd-sourced image segmentation. Our method involves training a regressor to estimate the quality of a segmentation from the annotator's clickstream data. The quality estimation can be used to identify spam and weight individual annotations by their (estimated) quality when merging multiple segmentations of one image. Using a total of 29,000 crowd annotations performed on publicly available data of different object classes, we show that (1) our method is highly accurate in estimating the segmentation quality based on clickstream data, (2) outperforms state-of-the-art methods for merging multiple annotations. As the regressor does not need to be trained on the object class that it is applied to it can be regarded as a low-cost option for quality control and confidence analysis in the context of crowd-based image annotation.

中文翻译：

---

自信地进行基于人群的对象分割的点击流分析


随着人们对基于机器学习的自动图像注释解决方案的兴趣迅速增加，用于算法训练的参考注释的可用性是该领域的主要瓶颈之一。众包已发展成为低成本和大规模数据注释的宝贵选择；然而，质量控制仍然是一个需要解决的主要问题。据我们所知，我们是第一个分析注释过程以改进众包图像分割的人。我们的方法涉及训练回归器来根据注释器的点击流数据估计分段的质量。质量估计可用于在合并一幅图像的多个分割时识别垃圾邮件并根据其（估计）质量对各个注释进行加权。通过对不同对象类的公开数据执行总共 29,000 个人群注释，我们表明 (1) 我们的方法在基于点击流数据估计分割质量方面非常准确，(2) 优于最先进的方法用于合并多个注释。由于回归器不需要对其所应用的对象类进行训练，因此可以将其视为基于人群的图像注释背景下质量控制和置信度分析的低成本选择。