Emotion recognition has attracted great interest. Numerous emotion recognition approaches have been proposed, most of which focus on visual, acoustic or psychophysiological information individually. Although more recent research has considered multimodal approaches, individual modalities are often combined only by simple fusion or are directly fused with deep learning networks at the feature level. In this paper, we propose an approach to training several specialist networks that employs deep learning techniques to fuse the features of individual modalities. This approach includes a multimodal deep belief network (MDBN), which optimizes and fuses unified psychophysiological features derived from the features of multiple psychophysiological signals, a bimodal deep belief network (BDBN) that focuses on representative visual features among the features of a video stream, and another BDBN that focuses on the high multimodal features in the unified features obtained from two modalities. Experiments are conducted on the BioVid Emo DB database and 80.89% accuracy is achieved, which outperforms the state-of-the-art approaches. The results demonstrate that the proposed approach can solve the problems of feature redundancy and lack of key features caused by multimodal fusion.

中文翻译：

---

在多种心理生理信号和视频中使用多模式深度学习进行情绪识别

情感识别引起了极大的兴趣。已经提出了许多情感识别方法，其中大多数专注于视觉，听觉或心理生理信息。尽管最近的研究已经考虑了多模式方法，但是单个模式通常仅通过简单融合来组合，或者在功能级别上直接与深度学习网络融合。在本文中，我们提出了一种训练几个专家网络的方法，该方法采用深度学习技术来融合各个模态的特征。这种方法包括一个多模式深度信任网络（MDBN），该网络可以优化和融合从多个心理生理信号的特征中得出的统一心理生理特征，一个双峰深度置信网络（BDBN）专注于视频流特征中的代表性视觉特征，另一个BDBN聚焦于从两种模态获得的统一特征中的高多峰特征。在BioVid Emo DB数据库上进行了实验，达到了80.89％的准确度，这优于最新方法。结果表明，该方法可以解决多模态融合引起的特征冗余和关键特征缺失的问题。