In an emergency room (ER) setting, stroke triage or screening is a common challenge. A quick CT is usually done instead of MRI due to MRI’s slow throughput and high cost. Clinical tests are commonly referred to during the process, but the misdiagnosis rate remains high. We propose a novel multimodal deep learning framework, DeepStroke, to achieve computer-aided stroke presence assessment by recognizing patterns of minor facial muscles incoordination and speech inability for patients with suspicion of stroke in an acute setting. Our proposed DeepStroke takes one-minute facial video data and audio data readily available during stroke triage for local facial paralysis detection and global speech disorder analysis. Transfer learning was adopted to reduce face-attribute biases and improve generalizability. We leverage a multi-modal lateral fusion to combine the low- and high-level features and provide mutual regularization for joint training. Novel adversarial training is introduced to obtain identity-free and stroke-discriminative features. Experiments on our video-audio dataset with actual ER patients show that DeepStroke outperforms state-of-the-art models and achieves better performance than both a triage team and ER doctors, attaining a 10.94% higher sensitivity and maintaining 7.37% higher accuracy than traditional stroke triage when specificity is aligned. Meanwhile, each assessment can be completed in less than six minutes, demonstrating the framework’s great potential for clinical translation.

在急诊室 (ER) 环境中，中风分类或筛查是一项常见挑战。由于 MRI 的吞吐量慢且成本高，通常会进行快速 CT 而非 MRI。在此过程中通常会提到临床测试，但误诊率仍然很高。我们提出了一种新的多模式深度学习框架DeepStroke，通过识别急性环境中疑似中风患者的轻微面部肌肉不协调和言语无能的模式来实现计算机辅助的中风存在评估。我们提出的DeepStroke在中风分诊期间获取一分钟的面部视频数据和音频数据，用于局部面部麻痹检测和全球语言障碍分析。采用迁移学习来减少人脸属性偏差并提高泛化性。我们利用多模态横向融合来结合低级和高级特征，并为联合训练提供相互正则化。引入了新的对抗训练以获得无身份和中风判别特征。我们对实际 ER 患者的视频-音频数据集进行的实验表明，DeepStroke优于最先进的模型，并且比分诊团队和 ER 医生的表现更好，在特异性保持一致时，灵敏度比传统的中风分诊高 10.94%，准确度高 7.37%。同时，每项评估可以在不到六分钟的时间内完成，展示了该框架在临床转化方面的巨大潜力。