The current practice for assessing neonatal postoperative pain relies on bedside caregivers. This practice is subjective, inconsistent, slow, and discontinuous. To develop a reliable medical interpretation, several automated approaches have been proposed to enhance the current practice. These approaches are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As pain is a multimodal emotion that is often expressed through multiple modalities, the multimodal assessment of pain is necessary especially in case of postoperative (acute prolonged) pain. Additionally, spatio-temporal analysis is more stable over time and has been proven to be highly effective at minimizing misclassification errors. In this paper, we present a novel multimodal spatio-temporal approach that integrates visual and vocal signals and uses them for assessing neonatal postoperative pain. We conduct comprehensive experiments to investigate the effectiveness of the proposed approach. We compare the performance of the multimodal and unimodal postoperative pain assessment, and measure the impact of temporal information integration. The experimental results, on a real-world dataset, show that the proposed multimodal spatio-temporal approach achieves the highest AUC (0.87) and accuracy (79%), which are on average 6.67% and 6.33% higher than unimodal approaches. The results also show that the integration of temporal information markedly improves the performance as compared to the non-temporal approach as it captures changes in the pain dynamic. These results demonstrate that the proposed approach can be used as a viable alternative to manual assessment, which would tread a path toward fully automated pain monitoring in clinical settings, point-of-care testing, and homes.

目前评估新生儿术后疼痛的做法依赖于床边护理人员。这种做法是主观的、不一致的、缓慢的和不连续的。为了开发可靠的医学解释，已经提出了几种自动化方法来增强当前的实践。这些方法是单峰的，主要侧重于评估新生儿程序性（急性）疼痛。由于疼痛是一种多模态情绪，通常通过多种模态表达，因此有必要对疼痛进行多模态评估，尤其是在术后（急性长期）疼痛的情况下。此外，时空分析随着时间的推移更加稳定，并且已被证明在最大限度地减少错误分类错误方面非常有效。在本文中，我们提出了一种新的多模式时空方法，它整合了视觉和声音信号，并使用它们来评估新生儿术后疼痛。我们进行了全面的实验来研究所提出方法的有效性。我们比较了多模式和单模式术后疼痛评估的性能，并衡量时间信息整合的影响。在真实世界数据集上的实验结果表明，所提出的多模态时空方法实现了最高的 AUC（0.87）和准确度（79%），平均比单模态方法高 6.67% 和 6.33%。结果还表明，与非时间方法相比，时间信息的整合显着提高了性能，因为它捕捉了疼痛动态的变化。