Studies of time-continuous human behavioral phenomena often rely on ratings from multiple annotators. Since the ground truth of the target construct is often latent, the standard practice is to use ad-hoc metrics (such as averaging annotator ratings). Despite being easy to compute, such metrics may not provide accurate representations of the underlying construct. In this paper, we present a novel method for modeling multiple time series annotations over a continuous variable that computes the ground truth by modeling annotator specific distortions. We condition the ground truth on a set of features extracted from the data and further assume that the annotators provide their ratings as modification of the ground truth, with each annotator having specific distortion tendencies. We train the model using an Expectation-Maximization based algorithm and evaluate it on a study involving natural interaction between a child and a psychologist, to predict confidence ratings of the children’s smiles. We compare and analyze the model against two baselines where: (i) the ground truth in considered to be framewise mean of ratings from various annotators and, (ii) each annotator is assumed to bear a distinct time delay in annotation and their annotations are aligned before computing the framewise mean.

中文翻译：

---

基于真实推理和失真的多个时间序列注释建模

对时间连续的人类行为现象的研究通常依赖于多个注释者的评级。由于目标构造的基本事实通常是潜在的，因此标准做法是使用临时指标（例如平均注释者评级）。尽管易于计算，但此类指标可能无法提供底层构造的准确表示。在本文中，我们提出了一种对连续变量上的多个时间序列注释进行建模的新方法，该方法通过对特定于注释者的失真进行建模来计算真实情况。我们根据从数据中提取的一组特征来确定地面实况，并进一步假设注释者提供他们的评级作为对地面实况的修改，每个注释者都有特定的失真趋势。我们使用基于期望最大化的算法训练模型，并在一项涉及儿童和心理学家之间自然互动的研究中对其进行评估，以预测儿童微笑的置信度。我们将模型与两个基线进行比较和分析，其中：（i）基本事实被认为是来自各种注释者的评分的框架平均值，（ii）假设每个注释者在注释中具有明显的时间延迟，并且它们的注释是对齐的在计算帧均值之前。