Emotion is precious, useful for many applications such as public opinion detection and psychological disease prediction. Emotion recognition on multimodal data has attracted extensive attention. Although modifying model structure or multimodal feature fusion methods have contributed a lot to emotion recognition, little attention is paid to mining implicit emotion relationship. In this article, implicit emotion relationship consists of emotion distribution, confusion, and transfer. Emotion distribution allows multiple emotions in one sample, while confusion and transfer imply the prediction confusion and bias. In order to mine implicit emotion relationship in multimodal data, this article employs three image and two text classification models to recognize emotions, respectively. Two prediction emotion synthesis methods (optimal prediction emotion synthesis and majority prediction emotion synthesis) are proposed to synthesize the outputs of multiple models. Based on the results of two emotion synthesis methods, emotion distribution on samples is obtained. Emotion confusion and transfer among different emotion samples are analyzed by relative entropy and Jensen–Shannon divergence. Implicit emotion relationship mining has potential not only in the interpretation of model performance, but also in guiding the development of emotion recognition as prior knowledge. Finally, we take topic scenario as an instance to mine implicit emotion relationships.

中文翻译：

---

基于多模态数据预测的最优和多数综合的隐式情感关系挖掘

情感是宝贵的，在舆论检测和心理疾病预测等许多应用中都有用。多模态数据的情感识别引起了广泛关注。尽管修改模型结构或多模态特征融合方法对情感识别做出了很大贡献，但很少有人关注挖掘隐含情感关系。在本文中，内隐情感关系包括情感分布、困惑和转移。情绪分布允许一个样本中存在多种情绪，而混淆和转移意味着预测的混淆和偏差。为了挖掘多模态数据中的隐含情感关系，本文分别采用三个图像和两个文本分类模型来识别情感。提出了两种预测情感合成方法（最优预测情感合成和多数预测情感合成）来合成多个模型的输出。基于两种情感合成方法的结果，得到样本上的情感分布。通过相对熵和 Jensen-Shannon 散度分析不同情绪样本之间的情绪混淆和转移。隐式情感关系挖掘不仅在解释模型性能方面具有潜力，而且在指导情感识别作为先验知识的发展方面具有潜力。最后，我们以主题场景为例，挖掘内隐情感关系。得到样本上的情绪分布。通过相对熵和 Jensen-Shannon 散度分析不同情绪样本之间的情绪混淆和转移。隐式情感关系挖掘不仅在解释模型性能方面具有潜力，而且在指导情感识别作为先验知识的发展方面具有潜力。最后，我们以主题场景为例，挖掘内隐情感关系。得到样本上的情绪分布。通过相对熵和 Jensen-Shannon 散度分析不同情绪样本之间的情绪混淆和转移。隐式情感关系挖掘不仅在解释模型性能方面具有潜力，而且在指导情感识别作为先验知识的发展方面具有潜力。最后，我们以主题场景为例，挖掘内隐情感关系。