In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model’s fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

在这项工作中，我们介绍了一种深度学习架构，用于评估来自 40 名癫痫患者的多模态脑电图 (EEG) 和功能性近红外光谱 (fNIRS) 记录。长短期记忆单元和卷积神经网络集成在多模式序列到序列自动编码器中。经过训练的神经网络通过从 EEG 全光谱和特定 EEG 频带中分层提取深度特征来预测来自 EEG 的 fNIRS 信号，无先验。结果表明，与其他频率包络相比，较高频率的 EEG 范围可预测 fNIRS 信号，其中伽马波段输入主导 fNIRS 预测。基于种子的功能连接验证了实验 fNIRS 和我们模型的 fNIRS 重建之间的相似模式。