Mental disorder is a serious public health concern that affects the life of millions of people throughout the world. Early diagnosis is essential to ensure timely treatment and to improve the well-being of those affected by a mental disorder. In this paper, we present a novel multimodal framework to perform mental disorder recognition from videos. The proposed approach employs a combination of audio, video and textual modalities. Using recurrent neural network architectures, we incorporate the temporal information in the learning process and model the dynamic evolution of the features extracted for each patient. For multimodal fusion, we propose an efficient late fusion strategy based on a simple feed-forward neural network that we call adaptive nonlinear judge classifier. We evaluate the proposed framework on two mental disorder datasets. On both, the experimental results demonstrate that the proposed framework outperforms the state-of-the-art approaches. We also study the importance of each modality for mental disorder recognition and infer interesting conclusions about the temporal nature of each modality. Our findings demonstrate that careful consideration of the temporal evolution of each modality is of crucial importance to accurately perform mental disorder recognition.

精神障碍是一种严重的公共卫生问题，影响着全世界数百万人的生活。早期诊断对于确保及时治疗和改善精神障碍患者的福祉至关重要。在本文中，我们提出了一种新的多模态框架来从视频中执行精神障碍识别。所提出的方法采用了音频、视频和文本形式的组合。使用循环神经网络架构，我们将时间信息纳入学习过程，并对为每个患者提取的特征的动态演变进行建模。对于多模态融合，我们提出了一种基于简单前馈神经网络的高效后期融合策略，我们称之为自适应非线性判断分类器. 我们在两个精神障碍数据集上评估了提议的框架。在这两个方面，实验结果表明所提出的框架优于最先进的方法。我们还研究了每种模式对精神障碍识别的重要性，并推断出有关每种模式的时间性质的有趣结论。我们的研究结果表明，仔细考虑每种模式的时间演变对于准确执行精神障碍识别至关重要。