A lot of research has been done on the detection of mental workload (MWL) using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of measurement modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify MWL. The goal of this research was to classify perceived mental workload (PMWL) using a deep neural network (DNN) that flexibly makes use of multiple modalities, in order to allow for feature sharing between modalities. To achieve this goal, an experiment was conducted in which MWL was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had 1 h to solve as many puzzles as they could. Between puzzles, they gave a difficulty rating between 1 and 7, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms and eye movements were collected simultaneously using LabStreamingLayer (LSL). Marker information from the puzzles was also streamed on LSL. We designed and evaluated a novel intermediate fusion multimodal DNN for the classification of PMWL using the aforementioned four modalities. Two main criteria that guided the design and implementation of our DNN are modularity and generalisability. We were able to classify PMWL within-level accurate (0.985 levels) on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available.

中文翻译：

---

使用中间融合多模态深度学习对感知心理工作量进行分类


关于使用各种生物信号检测脑力负荷（MWL）已经进行了大量研究。最近，深度学习带来了新颖的方法和结果。多种测量方式已被证明在这项任务中很有价值，但目前的研究通常仅使用单一方式对 MWL 进行分类。本研究的目标是使用灵活利用多种模式的深度神经网络 (DNN) 对感知脑力负荷 (PMWL) 进行分类，以便允许模式之间的特征共享。为了实现这一目标，我们进行了一项实验，在语言逻辑谜题的帮助下模拟 MWL。这些谜题有五个难度级别，并以随机顺序呈现。参与者有 1 小时的时间来解决尽可能多的谜题。在谜题之间，他们给出了 1 到 7 之间的难度等级，其中 7 是最高难度。使用 LabStreamingLayer (LSL) 同时收集皮肤电反应、光电体积描记图、功能性近红外光谱图和眼球运动。谜题中的标记信息也在 LSL 上进行传输。我们设计并评估了一种新型中间融合多模态 DNN，用于使用上述四种模态对 PMWL 进行分类。指导 DNN 设计和实现的两个主要标准是模块化和通用性。我们能够使用上述方式在七级工作负载量表上对 PMWL 进行级别内准确（0.985 级）分类。由于设计的模块化性质，模型架构允许轻松添加和删除模式，而不会产生重大结构影响。此外，我们表明，与单一模态相比，我们的神经网络在使用多种模态时表现更好。 本文使用的数据集和代码是公开可用的。