We proposed a probabilistic approach to joint modeling of participants’ reliability and humans’ regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming from a targeted population. Crowdsourcing-based studies or experiments, which rely on human self-reported affect, pose additional challenges as compared with typical crowdsourcing studies that attempt to acquire concrete non-affective labels of objects. The reliability of participants has been massively pursued for typical non-affective crowdsourcing studies, whereas the regularity of humans in an affective experiment in its own right has not been thoroughly considered. It has been often observed that different individuals exhibit different feelings on the same test question, which does not have a sole correct response in the first place. High reliability of responses from one individual thus cannot conclusively result in high consensus across individuals. Instead, globally testing consensus of a population is of interest to investigators. Built upon the agreement multigraph among tasks and workers, our probabilistic model differentiates subject regularity from population reliability. We demonstrate the method’s effectiveness for in-depth robust analysis of large-scale crowdsourced affective data, including emotion and aesthetic assessments collected by presenting visual stimuli to human subjects.

中文翻译：

---

用于提高众包情感数据质量的概率多图建模

我们提出了一种概率方法，用于在众包情感研究中对参与者的可靠性和人类的规律性进行联合建模。可靠性衡量受试者认真回答问题的可能性；规律性衡量一个人同意来自目标人群的其他认真输入的反应的频率。与试图获取对象的具体非情感标签的典型众包研究相比，基于众包的研究或实验依赖于人类自我报告的影响，带来了额外的挑战。典型的非情感众包研究已经大量追求参与者的可靠性，而人类在情感实验中的规律性本身并没有得到彻底的考虑。经常观察到，不同的人对同一个测试问题表现出不同的感受，首先没有唯一正确的答案。因此，来自一个人的反应的高可靠性不能最终导致个人之间的高度共识。取而代之的是，研究人员感兴趣的是对人群的全球测试共识。我们的概率模型基于任务和工人之间的一致性多重图，将主题规律性与总体可靠性区分开来。我们证明了该方法对大规模众包情感数据进行深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。因此，来自一个人的反应的高可靠性不能最终导致个人之间的高度共识。取而代之的是，研究人员感兴趣的是对人群的全球测试共识。我们的概率模型基于任务和工人之间的一致性多重图，将主题规律性与总体可靠性区分开来。我们证明了该方法对大规模众包情感数据进行深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。因此，来自一个人的反应的高可靠性不能最终导致个人之间的高度共识。取而代之的是，研究人员感兴趣的是对人群的全球测试共识。我们的概率模型基于任务和工人之间的一致性多重图，将主题规律性与总体可靠性区分开来。我们证明了该方法对大规模众包情感数据进行深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。我们的概率模型将主题规律性与总体可靠性区分开来。我们证明了该方法对大规模众包情感数据进行深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。我们的概率模型将主题规律性与总体可靠性区分开来。我们证明了该方法对大规模众包情感数据进行深入稳健分析的有效性，包括通过向人类受试者呈现视觉刺激而收集的情感和审美评估。