EEG-based approach for recognizing human social emotion perception

Abstract

Social emotion perception plays an important role in our daily social interactions and is involved in the treatments for mental disorders. Hyper-scanning technique enables to measure brain activities simultaneously from two or more persons, which was employed in this study to explore social emotion perception. We analyzed the recorded electroencephalogram (EEG) to explore emotion perception in terms of event related potential (ERP) and phase synchronization, and classified emotion categories based on convolutional neural network (CNN). The results showed that (1) ERP was significantly different among four emotion categories (i.e., anger, disgust, neutral, and happy), but there was no significant difference for ERP in the comparison of rating orders (the order of rating actions of the paired participants); (2) the intra-brain phase lag index (PLI) was higher than the inter-brain PLI but its number of connections exhibiting significant difference was less in all typical frequency bands (from delta to gamma); (3) the emotion classification accuracy of inter-PLI-Conv outperformed that of intra-PLI-Conv for all cases of using each frequency band (five frequency bands totally). In particular, the classification accuracies averaged across all participants in the alpha band were 65.55% and 50.77% (much higher than the chance level) for the inter-PLI-Conv and intra-PLI-Conv, respectively. According to our results, the emotion category of happiness can be classified with a higher performance compared to the other categories.

Introduction

It is undoubted that emotion is crucial component during human–human interaction and human–machine interaction in our everyday life [1]. Emotion is also relevant to some mental diseases and detrimental habits such as internet addiction, tristimania [2], anxiety and social phobia [3]. Although a great deal of efforts have been made by researchers from diverse disciplines (e.g., neuroscience, psychology, and computer science) to investigate emotion perception, the knowledge we currently acquired is still limited [4], [5]. Most of the prior studies explored emotion perception in a scenario of one person [6], [7], [8], [9], rather than multiple persons who interact each other. In our daily life, people usually perceive emotion during interacting with others [10]. For instance, how you feel yourself and other people during the social interaction [11], [12]. Therefore, it is better to investigate emotion perception during social interaction.

In recent years, hyper-scanning, a technique for measuring brain activities simultaneously from two or more persons, has been utilized to explore brain-to-brain interactions when two or more persons engage in a task [13]. The hyper-scanning could be categorized as fNIRS (functional near infrared spectroscopy) hyper-scanning [14], fMRI (functional magnetic resonance imaging) hyper-scanning [15] and EEG hyper-scanning [16] according to the signal recorded in the experiment. Because EEG is of high temporal resolution and ease of use with an advantage of low cost [17], we employed EEG hyper-scanning in our study. Previous studies revealed that the frontal region was dominantly involved in the emotion recognition, especially facial expression recognition [18], [19], [20]. This hints us to focus on the frontal region for emotion perception exploration in the context of multiple persons. The concentration on a particular region could also benefit the following step of emotion classification as less amount of data are required for processing.

In this study, we designed an EEG hyper-scanning experiment to explore emotion perception in the scenario of two-person interaction. The characteristics of brain activities were first explored to reveal neural mechanisms related to emotion perception. Then, a deep learning model was utilized to classify different emotion categories based on the findings in the data analysis. The framework of our study was depicted in Fig. 1. The main contributions of this paper are (1) designing an EEG hyper-scanning experiment to explore emotion perception in the scenario of two-person interaction; (2) revealing EEG characteristics associated with emotion perception; (3) evaluating the effectiveness of intra-brain and inter-brain phase synchronization features; (4) recognizing emotions based on EEG signal only using twelve electrodes on the frontal region, which could facilitate the development of portable emotion recognition system. The remainder of the paper is organized as follows. The relevant work is introduced in Section 2, which is followed by the description of experiment design and setup in Section 3. Next, the methodological descriptions of the preprocessing, ERP, synchronization feature extraction, convolutional neural networks are given in Section 4, which is followed by the results in Section 5. Finally, conclusions are drawn in Section 6.