Emotion recognition is an attractive and essential topic in image and signal processing. In this paper, we propose a multi-level fusion method to combine visual information and physiological signals for emotion recognition. For visual information, we propose a serial fusion of two-stage features to enhance the representation of facial expression in a video sequence. We propose to integrate the Neural Aggregation Network with Convolutional Neural Network feature map to reinforce the vital emotional frames. For physiological signals, we propose a parallel fusion scheme to widen the band of the annotation of the electroencephalogram signals. We extract the frequency feature with the Linear-Frequency Cepstral Coefficients and enhance it with the signal complexity denoted by Sample Entropy (SampEn). In the classification stage, we realize both feature level and decision level fusion of both visual and physiological information. Experimental results validate the effectiveness of the proposed multi-level multi-modal feature representation method.

情感识别是图像和信号处理中一个有吸引力且必不可少的主题。在本文中，我们提出了一种多层次融合方法，将视觉信息和生理信号相结合以进行情感识别。对于视觉信息，我们提出了两阶段特征的串行融合，以增强视频序列中面部表情的表示。我们建议将神经聚合网络与卷积神经网络特征图进行集成，以增强重要的情感框架。对于生理信号，我们提出了一种并行融合方案以加宽脑电图信号的注释范围。我们使用线性频率倒谱系数提取频率特征，并通过采样熵（SampEn）表示的信号复杂度来增强频率特征。在分类阶段 我们实现视觉和生理信息的特征级和决策级融合。实验结果验证了所提出的多级多模式特征表示方法的有效性。