The assessment of gaze behaviour is essential for understanding the psychology of communication. Mobile eye-tracking glasses are useful to measure gaze behaviour during dynamic interactions. Eye-tracking data can be analysed by using manually annotated areas-of-interest. Computer vision algorithms may alternatively be used to reduce the amount of manual effort, but also the subjectivity and complexity of these analyses. Using additional re-identification (Re-ID) algorithms, different participants in the interaction can be distinguished. The aim of this study was to compare the results of manual annotation of mobile eye-tracking data with the results of a computer vision algorithm. We selected the first minute of seven randomly selected eye-tracking videos of consultations between physicians and patients in a Dutch Internal Medicine out-patient clinic. Three human annotators and a computer vision algorithm annotated mobile eye-tracking data, after which interrater reliability was assessed between the areas-of-interest annotated by the annotators and the computer vision algorithm. Additionally, we explored interrater reliability when using lengthy videos and different area-of-interest shapes. In total, we analysed more than 65 min of eye-tracking videos manually and with the algorithm. Overall, the absolute normalized difference between the manual and the algorithm annotations of face-gaze was less than 2%. Our results show high interrater agreements between human annotators and the algorithm with Cohen’s kappa ranging from 0.85 to 0.98. We conclude that computer vision algorithms produce comparable results to those of human annotators. Analyses by the algorithm are not subject to annotator fatigue or subjectivity and can therefore advance eye-tracking analyses.

凝视行为的评估对于理解交流心理学至关重要。移动眼动追踪眼镜可用于测量动态交互过程中的注视行为。可以通过使用手动注释的感兴趣区域来分析眼动追踪数据。计算机视觉算法也可用于减少人工工作量，以及这些分析的主观性和复杂性。使用额外的重新识别 (Re-ID) 算法，可以区分交互中的不同参与者。本研究的目的是将移动眼动追踪数据的手动注释结果与计算机视觉算法的结果进行比较。我们选择了荷兰内科门诊中医生和患者之间会诊的七个随机选择的眼动追踪视频的第一分钟。三个人工注释器和一个计算机视觉算法对移动眼动追踪数据进行注释，然后在注释器注释的感兴趣区域和计算机视觉算法之间评估评估者间的可靠性。此外，我们在使用冗长的视频和不同的兴趣区域形状时探索了评价者之间的可靠性。总的来说，我们手动和使用算法分析了超过 65 分钟的眼动追踪视频。总体而言，人脸注视的手动标注和算法标注之间的绝对归一化差异小于 2%。我们的结果显示人类注释者与 Cohen 的 kappa 范围从 0.85 到 0.98 的算法之间的高一致性。我们得出结论，计算机视觉算法产生的结果与人类注释者的结果相当。该算法的分析不受注释者疲劳或主观性的影响，因此可以推进眼动追踪分析。