This paper presents the extraction of the emotional signals from traumatic brain-injured (TBI) patients through the analysis of facial features and implementation of the effective emotion-recognition model through the Pepper robot to assist in the rehabilitation process. The identification of emotional cues from TBI patients is very challenging due to unique and diverse psychological, physiological, and behavioral challenges such as non-cooperation, facial/body paralysis, upper or lower limb impairments, cognitive, motor, and hearing skills inhibition. It is essential to read subtle changes in the emotional cues of TBI patients for effective communication and the development of affect-based systems. To analyze the variations of the emotional signal in TBI patients, a new database is collected in a natural and unconstrained environment from eleven residents of a neurological center in three different modalities, RGB, thermal and depth in three specified scenarios performing physical, cognitive and social communication rehabilitation activities. Due to the lack of labeled data, a deep transfer learning method is applied to efficiently classify emotions. The emotion classification model is tested through closed-field study and installment of a Pepper robot equipped with the trained model. Our deep trained and fine-tuned emotional recognition model composed of CNN-LSTM has improved the performance by 1.47% on MMI, and 4.96% on FER2013 validation data set. In addition, use of temporal information and transfer learning techniques to overcome TBI-data limitations has increased the performance efficacy on challenging dataset of neurologically impaired people. Findings that emerged from the study illustrate the noticeable effectiveness of SoftBank Pepper robot equipped with deep trained emotion recognition model in developing rehabilitation strategies by monitoring the TBI patient’s emotions. This research article presents the technical solution for real therapeutic robot interaction to rehabilitate patients with standard monitoring, assessment, and feedback in the neuro centers.

本文介绍了通过面部特征分析提取脑外伤（TBI）患者的情绪信号，并通过 Pepper 机器人实施有效的情绪识别模型来辅助康复过程。由于独特且多样的心理、生理和行为挑战，例如不合作、面部/身体麻痹、上肢或下肢障碍、认知、运动和听力技能抑制，识别来自 TBI 患者的情绪线索非常具有挑战性。阅读 TBI 患者情绪线索的细微变化对于有效沟通和基于情感的系统的发展至关重要。为了分析 TBI 患者情绪信号的变化，一个新的数据库是在自然和不受约束的环境中从神经中心的 11 位居民以三种不同的方式收集的，RGB、热和深度在三个特定场景中进行身体、认知和社会交流康复活动。由于缺乏标记数据，应用深度迁移学习方法对情绪进行有效分类。情感分类模型通过闭场研究和配备训练模型的 Pepper 机器人的安装进行测试。我们由 CNN-LSTM 组成的经过深度训练和微调的情感识别模型在 MMI 上提高了 1.47% 的性能，在 FER2013 验证数据集上提高了 4.96%。此外，使用时间信息和转移学习技术来克服 TBI 数据限制提高了对神经受损人群具有挑战性的数据集的性能效率。研究结果表明，配备经过深度训练的情绪识别模型的 SoftBank Pepper 机器人在通过监测 TBI 患者的情绪来制定康复策略方面具有显着的有效性。这篇研究文章提出了真实治疗机器人交互的技术解决方案，以在神经中心通过标准的监测、评估和反馈来使患者康复。研究结果表明，配备经过深度训练的情绪识别模型的 SoftBank Pepper 机器人在通过监测 TBI 患者的情绪来制定康复策略方面具有显着的有效性。这篇研究文章提出了真实治疗机器人交互的技术解决方案，以在神经中心通过标准的监测、评估和反馈来使患者康复。研究结果表明，配备经过深度训练的情绪识别模型的 SoftBank Pepper 机器人在通过监测 TBI 患者的情绪来制定康复策略方面具有显着的有效性。这篇研究文章提出了真实治疗机器人交互的技术解决方案，以在神经中心通过标准的监测、评估和反馈来使患者康复。