In this article we propose the use of convolutional self-attention for attention-based representation learning, while replacing traditional vectorization methods with a transformer as the backbone of our speech model for transfer learning within our automatic deceit detection framework. This design performs a multimodal data analysis and applies fusion to merge visual, vocal, and speech(textual) channels; reporting deceit predictions. Our experimental results show that the proposed architecture improves the state-of-the-art on the popular Real-Life Trial (RLT) dataset in terms of correct classification rate. To further assess the generalizability of our design, we experiment on the low-stakes Box of Lies (BoL) dataset and achieve state-of-the-art performance as well as providing cross-corpus comparisons. Following our analysis, we report that (1) convolutional self-attention learns meaningful representations while performing joint attention computation for deception, (2) apparent deceptive intent is a continuous function of time and subjects can display varying levels of apparent deceptive intent throughout recordings, and (3), in support of criminal psychology findings, studying abnormal behavior out of context can be an unreliable way to predict deceptive intent.

中文翻译：

---

通过高风险法庭审判的多模态分析自动检测欺骗行为

在本文中，我们建议使用卷积自注意力进行基于注意力的表示学习，同时用变压器替换传统的向量化方法，作为我们的语音模型的骨干，在我们的自动欺骗检测框架内进行迁移学习。该设计执行多模态数据分析，并应用融合来合并视觉、声音和语音（文本）通道；报告欺骗性预测。我们的实验结果表明，所提出的架构提高了流行的最新技术真实生活试验 (RLT) 数据集的正确分类率。为了进一步评估我们设计的普遍性，我们在低风险的情况下进行了实验Box of Lies (BoL) 数据集，实现最先进的性能并提供跨语料库比较。根据我们的分析，我们报告说（1）卷积自注意力学习有意义的表示，同时对欺骗进行联合注意力计算，（2）明显的欺骗意图是时间的连续函数，并且受试者可以在整个录音中表现出不同程度的明显欺骗意图， (3)，为了支持犯罪心理学的发现，断章取义地研究异常行为可能是预测欺骗意图的不可靠方法。