Measuring airways in chest computed tomography (CT) scans is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated scans for good performance. We investigate whether crowdsourcing can be used to gather airway annotations. We generate image slices at known locations of airways in 24 subjects and request the crowd workers to outline the airway lumen and airway wall. After combining multiple crowd workers, we compare the measurements to those made by the experts in the original scans. Similar to our preliminary study, a large portion of the annotations were excluded, possibly due to workers misunderstanding the instructions. After excluding such annotations, moderate to strong correlations with the expert can be observed, although these correlations are slightly lower than inter-expert correlations. Furthermore, the results across subjects in this study are quite variable. Although the crowd has potential in annotating airways, further development is needed for it to be robust enough for gathering annotations in practice. For reproducibility, data and code are available online: \url{http://github.com/adriapr/crowdairway.git}.

中文翻译：

---

胸部计算机断层扫描图像中的众包气道注释

在胸部计算机断层扫描（CT）扫描中测量气道对于表征疾病如囊性纤维化非常重要，但手动执行非常耗时。机器学习算法提供了一种替代方法，但需要大量带注释的扫描才能获得良好的性能。我们调查众包是否可用于收集气道注释。我们在24个对象的气道已知位置生成图像切片，并要求人群工作者勾勒出气道内腔和气道壁轮廓。在组合了多个人群工作者之后，我们将测量结果与专家在原始扫描中所做的测量结果进行比较。与我们的初步研究类似，很大一部分注释被排除在外，可能是由于工人误解了说明。排除此类注释后，可以观察到与专家的中等至强相关性，尽管这些相关性比专家间的相关性略低。此外，本研究中各科目的结果差异很大。尽管人群具有对气道进行注释的潜力，但仍需要进一步开发，使其足够健壮以在实践中收集注释。为了重现性，可在线获取数据和代码：\ url {http://github.com/adriapr/crowdairway.git}。