Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.

中文翻译：

---

FusionSense：在受大脑启发的尖峰神经网络中使用多模态数据和深度学习的特征融合进行情感分类。


利用多模态信号解决情感识别问题是情感计算的新兴趋势之一。多项研究利用最先进的深度学习方法，结合生理信号，如心电图 (EEG)、脑电图 (ECG)、皮肤温度以及面部表情、声音、姿势等，对情绪进行分类。尖峰神经网络 (SNN) 代表第三代神经网络，并采用生物学上合理的神经元模型。 SNN 已被证明能够以有效的方式处理时空数据，这本质上是情感识别问题中遇到的数据的本质。在这项工作中，我们首次提出应用 SNN 来解决多模态数据集的情感识别问题。具体来说，我们使用 NeuCube 框架，该框架采用不断发展的 SNN 架构来对情绪效价进行分类并评估我们的方法在 MAHNOB-HCI 数据集上的性能。我们工作中使用的多模态数据包括面部表情以及心电图、皮肤温度、皮肤电导、呼吸信号、嘴巴长度和瞳孔大小等生理信号。我们在留一主题排除（LOSO）交叉验证模式下进行分类。我们的结果表明，在应用特征级融合时，所提出的方法对二元价进行分类的准确率达到了 73.15%，与其他深度学习方法相当。即使不使用脑电图，我们也能达到这种准确性，而其他深度学习方法则依赖脑电图来达到这种准确性。 总之，我们证明了 SNN 可以成功地用于解决多模态数据的情感识别问题，并为未来利用 SNN 进行情感计算的研究提供了方向。除了良好的准确性之外，SNN 识别系统还需要以自适应方式对新数据进行增量训练。它只需一次培训，这使得它适合实际和在线应用。这些特点是其他解决该问题的方法所没有体现出来的。