Emotion recognition is a hot research in modern intelligent systems. The technique is pervasively used in autonomous vehicles, remote medical service, and human–computer interaction (HCI). Traditional speech emotion recognition algorithms cannot be effectively generalized since both training and testing data are from the same domain, which have the same data distribution. In practice, however, speech data is acquired from different devices and recording environments. Thus, the data may differ significantly in terms of language, emotional types and tags. To solve such problem, in this work, we propose a bimodal fusion algorithm to realize speech emotion recognition, where both facial expression and speech information are optimally fused. We first combine the CNN and RNN to achieve facial emotion recognition. Subsequently, we leverage the MFCC to convert speech signal to images. Therefore, we can leverage the LSTM and CNN to recognize speech emotion. Finally, we utilize the weighted decision fusion method to fuse facial expression and speech signal to achieve speech emotion recognition. Comprehensive experimental results have demonstrated that, compared with the uni-modal emotion recognition, bimodal features-based emotion recognition achieves a better performance.

情感识别是现代智能系统中的热门研究。该技术广泛用于自动驾驶汽车，远程医疗服务和人机交互（HCI）。传统的语音情感识别算法无法有效地推广，因为训练和测试数据都来自同一域，具有相同的数据分布。然而，实际上，语音数据是从不同的设备和记录环境中获取的。因此，数据在语言，情感类型和标签方面可能存在显着差异。为了解决这个问题，在这项工作中，我们提出了一种双峰融合算法来实现语音情感识别，其中面部表情和语音信息都得到了最佳融合。我们首先将CNN和RNN结合起来以实现面部表情识别。后来，我们利用MFCC将语音信号转换为图像。因此，我们可以利用LSTM和CNN识别语音情感。最后，利用加权决策融合方法融合表情和语音信号，实现语音情感识别。综合实验结果表明，与单峰情感识别相比，基于双峰特征的情感识别具有更好的性能。