Depression is a severe mental illness that impairs a person's capacity to function normally in personal and professional life. The assessment of depression usually requires a comprehensive examination by an expert professional. Recently, machine learning-based automatic depression assessment has received considerable attention for a reliable and efficient depression diagnosis. Various techniques for automated depression detection were developed; however, certain concerns still need to be investigated. In this work, we propose a novel deep multi-modal framework that effectively utilizes facial and verbal cues for an automated depression assessment. Specifically, we first partition the audio and video data into fixed-length segments. Then, these segments are fed into the Spatio-Temporal Networks as input, which captures both spatial and temporal features as well as assigns higher weights to the features that contribute most. In addition, Volume Local Directional Structural Pattern (VLDSP) based dynamic feature descriptor is introduced to extract the facial dynamics by encoding the structural aspects. Afterwards, we employ the Temporal Attentive Pooling (TAP) approach to summarize the segment-level features for audio and video data. Finally, the multi-modal factorized bilinear pooling (MFB) strategy is applied to fuse the multi-modal features effectively. An extensive experimental study reveals that the proposed method outperforms state-of-the-art approaches.

中文翻译：

---

基于深度多模态网络的自动抑郁严重程度估计


抑郁症是一种严重的精神疾病，会损害一个人在个人和职业生活中正常运作的能力。抑郁症的评估通常需要由专业人士进行全面检查。最近，基于机器学习的自动抑郁症评估因其可靠、高效的抑郁症诊断而受到广泛关注。开发了各种自动抑郁症检测技术；然而，某些问题仍需要调查。在这项工作中，我们提出了一种新颖的深层多模式框架，该框架有效地利用面部和言语线索进行自动抑郁症评估。具体来说，我们首先将音频和视频数据划分为固定长度的段。然后，这些片段作为输入输入时空网络，该网络捕获空间和时间特征，并为贡献最大的特征分配更高的权重。此外，引入基于体积局部定向结构模式（VLDSP）的动态特征描述符，通过对结构方面进行编码来提取面部动态。然后，我们采用时间注意力池（TAP）方法来总结音频和视频数据的分段级特征。最后，应用多模态分解双线性池（MFB）策略有效地融合多模态特征。广泛的实验研究表明，所提出的方法优于最先进的方法。