Emotion recognition is a crucial application in human–computer interaction. It is usually conducted using facial expressions as the main modality, which might not be reliable. In this study, we proposed a multimodal approach that uses 2-channel electroencephalography (EEG) signals and eye modality in addition to the face modality to enhance the recognition performance. We also studied the use of facial images versus facial depth as the face modality and adapted the common arousal–valence model of emotions and the convolutional neural network, which can model the spatiotemporal information from the modality data for emotion recognition. Extensive experiments were conducted on the modality and emotion data, the results of which showed that our system has high accuracies of 67.8% and 77.0% in valence recognition and arousal recognition, respectively. The proposed method outperformed most state-of-the-art systems that use similar but fewer modalities. Moreover, the use of facial depth has outperformed the use of facial images. The proposed method of emotion recognition has significant potential for integration into various educational applications.

情感识别是人机交互的重要应用。它通常以面部表情为主要方式进行，这可能不可靠。在这项研究中，我们提出了一种多模态方法，除了面部模态外，还使用 ​​2 通道脑电图 (EEG) 信号和眼睛模态来提高识别性能。我们还研究了面部图像与面部深度作为面部模态的使用，并采用了常见的情绪唤醒 - 效价模型和卷积神经网络，它可以对来自模态数据的时空信息进行建模以进行情绪识别。对模态和情感数据进行了大量实验，结果表明我们的系统在价识别和唤醒识别方面的准确率分别为 67.8% 和 77.0%。所提出的方法优于大多数使用相似但模态较少的最先进系统。此外，面部深度的使用优于面部图像的使用。所提出的情绪识别方法具有集成到各种教育应用中的巨大潜力。