This paper reports on research to design an ensemble deep learning framework that integrates fine-tuned, three-stream hybrid deep neural network (i.e., Ensemble Deep Learning Model, EDLM), employing Convolutional Neural Network (CNN) to extract facial image features, detect and accurately classify the pain. To develop the approach, the VGGFace is fine-tuned and integrated with Principal Component Analysis and employed to extract features in images from the Multimodal Intensity Pain database at the early phase of the model fusion. Subsequently, a late fusion, three layers hybrid CNN and recurrent neural network algorithm is developed with their outputs merged to produce image-classified features to classify pain levels. The EDLM model is then benchmarked by means of a single-stream deep learning model including several competing models based on deep learning methods. The results obtained indicate that the proposed framework is able to outperform the competing methods, applied in a multi-level pain detection database to produce a feature classification accuracy that exceeds 89 %, with a receiver operating characteristic of 93 %. To evaluate the generalization of the proposed EDLM model, the UNBC-McMaster Shoulder Pain dataset is used as a test dataset for all of the modelling experiments, which reveals the efficacy of the proposed method for pain classification from facial images. The study concludes that the proposed EDLM model can accurately classify pain and generate multi-class pain levels for potential applications in the medical informatics area, and should therefore, be explored further in expert systems for detecting and classifying the pain intensity of patients, and automatically evaluating the patients’ pain level accurately.

本文报告了设计集成深度学习框架的研究，该框架集成了微调的三流混合深度神经网络（即., Ensemble Deep Learning Model, EDLM)，采用卷积神经网络 (CNN) 提取面部图像特征，检测并准确分类疼痛。为了开发该方法，VGGFace 与主成分分析进行了微调和集成，并用于在模型融合的早期阶段从多模态强度疼痛数据库中提取图像特征。随后，开发了一种后期融合、三层混合 CNN 和递归神经网络算法，将它们的输出合并以产生图像分类特征以对疼痛级别进行分类。然后通过单流深度学习模型对 EDLM 模型进行基准测试，该模型包括多个基于深度学习方法的竞争模型。获得的结果表明，所提出的框架能够胜过竞争方法，应用于多级疼痛检测数据库，产生超过 89% 的特征分类准确率，接收器操作特性为 93%。为了评估所提出的 EDLM 模型的泛化，UNBC-McMaster 肩痛数据集被用作所有建模实验的测试数据集，这揭示了所提出的面部图像疼痛分类方法的有效性。该研究得出的结论是，所提出的 EDLM 模型可以准确地对疼痛进行分类并生成多级疼痛等级，以用于医学信息学领域的潜在应用，因此，应在专家系统中进一步探索，以检测和分类患者的疼痛强度，并自动准确评估患者的疼痛程度。