Multimodal affect recognition constitutes an important aspect for enhancing interpersonal relationships in human-computer interaction. However, relevant data is hard to come by and notably costly to annotate, which poses a challenging barrier to build robust multimodal affect recognition systems. Models trained on these relatively small datasets tend to overfit and the improvement gained by using complex state-of-the-art models is marginal compared to simple baselines. Meanwhile, there are many different multimodal affect recognition datasets, though each may be small. In this paper, we propose to leverage these datasets using weakly-supervised multi-task learning to improve the generalization performance on each of them. Specifically, we explore three multimodal affect recognition tasks: 1) emotion recognition; 2) sentiment analysis; and 3) sarcasm recognition. Our experimental results show that multi-tasking can benefit all these tasks, achieving an improvement up to 2.9% accuracy and 3.3% F1-score. Furthermore, our method also helps to improve the stability of model performance. In addition, our analysis suggests that weak supervision can provide a comparable contribution to strong supervision if the tasks are highly correlated.

中文翻译：

---

弱监督多任务学习的多模态情感识别

多峰影响识别是增强人机交互中人际关系的重要方面。但是，相关数据很难获得，而且注释成本很高，这对构建健壮的多模式影响识别系统构成了挑战。在这些相对较小的数据集上训练的模型往往过拟合，并且与简单基准相比，使用复杂的最新模型所获得的改进是微不足道的。同时，尽管每个模式可能很小，但有许多不同的多峰影响识别数据集。在本文中，我们建议利用弱监督多任务学习来利用这些数据集，以提高每个数据集的泛化性能。具体来说，我们探讨了三种多模式影响识别任务：1）情绪识别；2）情绪分析；3）讽刺识别。我们的实验结果表明，多任务处理可以使所有这些任务受益，将准确性提高了2.9％，将F1评分提高了3.3％。此外，我们的方法还有助于提高模型性能的稳定性。此外，我们的分析表明，如果任务高度相关，则弱监督可以为强监督提供类似的贡献。