In mental health assessment, it is validated that nonverbal cues like facial expressions can be indicative of depressive disorders. Recently, the multimodal fusion of facial appearance and dynamics based on convolutional neural networks has demonstrated encouraging performance in depression analysis. However, correlation and complementarity between different visual modalities have not been well studied in prior methods. In this paper, we propose a sequential fusion method for facial depression recognition. For mining the correlated and complementary depression patterns in multimodal learning, a chained-fusion mechanism is introduced to jointly learn facial appearance and dynamics in a unified framework. We show that such sequential fusion can provide a probabilistic perspective of the model correlation and complementarity between two different data modalities for improved depression recognition. Results on a benchmark dataset show the superiority of our method against several state-of-the-art alternatives.

在心理健康评估中，已证实面部表情等非语言线索可以指示抑郁症。最近，基于卷积神经网络的面部外观和动力学的多模态融合在抑郁症分析中表现出令人鼓舞的表现。然而，在先前的方法中还没有很好地研究不同视觉模式之间的相关性和互补性。在本文中，我们提出了一种用于面部抑郁识别的顺序融合方法。为了挖掘多模态学习中相关和互补的抑郁模式，引入了一种链式融合机制，在统一框架中联合学习面部外观和动态。我们表明，这种顺序融合可以提供两种不同数据模式之间模型相关性和互补性的概率视角，以改善抑郁症的识别。基准数据集的结果显示了我们的方法相对于几种最先进的替代方案的优越性。