Signatures have long been considered to be one of the most accepted and practical means of user verification, despite being vulnerable to skilled forgers. In contrast, EEG signals have more recently been shown to be more difficult to replicate, and to provide better biometric information in response to known a stimulus. In this paper, we propose combining these two biometric traits using a multimodal Siamese Neural Network (mSNN) for improved user verification. The proposed mSNN network learns discriminative temporal and spatial features from the EEG signals using an EEG encoder and from the offline signatures using an image encoder. Features of the two encoders are fused into a common feature space for further processing. A Siamese network then employs a distance metric based on the similarity and dissimilarity of the input features to produce the verification results. The proposed model is evaluated on a dataset of 70 users, comprised of 1400 unique samples. The novel mSNN model achieves a 98.57% classification accuracy with a 99.29% True Positive Rate (TPR) and False Acceptance Rate (FAR) of 2.14%, outperforming the current state-of-the-art by 12.86% (in absolute terms). This proposed network architecture may also be applicable to the fusion of other neurological data sources to build robust biometric verification or diagnostic systems with limited data size.

尽管签名容易受到熟练的伪造者的攻击，但签名一直被认为是用户接受的最实用的验证方法之一。相反，近来已显示出EEG信号更难以复制，并且响应于已知刺激而提供更好的生物统计信息。在本文中，我们建议使用多模式暹罗神经网络（mSNN）结合这两个生物特征，以改善用户验证。提出的mSNN网络使用EEG编码器从EEG信号中学习判别性时空特征，并使用图像编码器从离线签名中学习判别性时空特征。将两个编码器的特征融合到一个公共特征空间中，以进行进一步处理。然后，一个暹罗网络根据输入特征的相似性和不相似性采用距离度量，以产生验证结果。该模型在70个用户的数据集上进行了评估，该数据集包含1400个唯一样本。新颖的mSNN模型可实现98.57％的分类准确率，正确率（TPR）为99.29％，错误接受率（FAR）为2.14％，比目前的最新水平高出12.86％（绝对值）。所提出的网络体系结构还可适用于其他神经系统数据源的融合，以建立具有有限数据大小的健壮的生物特征验证或诊断系统。29％的真实肯定率（TPR）和错误接受率（FAR）为2.14％，比目前的最新水平高出12.86％（绝对值）。所提出的网络体系结构还可适用于其他神经系统数据源的融合，以建立具有有限数据大小的健壮的生物特征验证或诊断系统。29％的真实肯定率（TPR）和错误接受率（FAR）为2.14％，比目前的最新水平高出12.86％（绝对值）。所提出的网络体系结构还可适用于其他神经系统数据源的融合，以建立具有有限数据大小的健壮的生物特征验证或诊断系统。