Medical & Biological Engineering & Computing ( IF 2.6 ) Pub Date : 2021-05-24 , DOI: 10.1007/s11517-021-02383-1 Duc-Phong Nguyen 1 , Marie-Christine Ho Ba Tho 1 , Tien-Tuan Dao 2
|
Facial expression recognition plays an essential role in human conversation and human–computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.
Graphical abstract
中文翻译：
使用 3D 点集和几何深度学习增强面部表情识别
面部表情识别在人类对话和人机交互中起着至关重要的作用。先前的研究已经识别出主要基于需要敏感特征工程和传统机器学习方法的 2D 图像处理的面部表情。本研究的目的是通过直接在 3D 点云数据上应用一种称为几何深度学习的新型深度学习来识别面部表情。使用了两个数据库（博斯普鲁斯海峡和 SIAT-3DFE）。博斯普鲁斯海峡数据库包括 65 个主题，具有七种基本表情（即愤怒、厌恶、恐惧、快乐、悲伤、惊讶和中性）。SIAT-3DFE 数据库有 150 个主题和 4 种基本面部表情（中性、快乐、悲伤和惊讶）。一、人脸中心裁剪等预处理程序，数据增强和点云去噪应用于 3D 面部扫描。然后，应用了名为 PointNet++ 的几何深度学习模型。执行超参数调整过程以找到最佳模型参数。最后，使用识别率和混淆矩阵对开发的模型进行评估。Bosphorus数据库中7种表情的面部表情识别准确率为69.01%，识别5种特定表情（愤怒、厌恶、高兴、惊讶和中性）时可达85.85%。SIAT-3DFE数据库的识别率为78.70%。目前的研究表明，可以使用几何深度学习方法直接处理 3D 点云以进行面部表情识别。在观点上，
京公网安备 11010802027423号