Brain decoding has shown that viewed image categories can be estimated from evoked functional magnetic resonance imaging (fMRI) activity. Recent studies attempted to estimate viewed image categories that were not used for training previously. Nevertheless, the estimation performance is limited since it is difficult to collect a large amount of fMRI data for training. This paper presents a method to accurately estimate viewed image categories not used for training via a semi-supervised multi-view Bayesian generative model. Our model focuses on the relationship between fMRI activity and multiple modalities, i.e., visual features extracted from viewed images and semantic features obtained from viewed image categories. Furthermore, in order to accurately estimate image categories not used for training, our semi-supervised framework incorporates visual and semantic features obtained from additional image categories in addition to image categories of training data. The estimation performance of the proposed model outperforms existing state-of-the-art models in the brain decoding field and achieves more than 95% identification accuracy. The results also have shown that the incorporation of additional image category information is remarkably effective when the number of training samples is small. Our semi-supervised framework is significant for the brain decoding field where brain activity patterns are insufficient but visual stimuli are sufficient.

中文翻译：

---

通过半监督多视图贝叶斯生成模型对查看图像类别进行大脑解码

大脑解码表明，可以从诱发的功能磁共振成像 (fMRI) 活动中估计查看的图像类别。最近的研究试图估计以前未用于训练的查看图像类别。然而，由于难以收集大量 fMRI 数据进行训练，因此估计性能是有限的。本文提出了一种通过半监督多视图贝叶斯生成模型准确估计未用于训练的查看图像类别的方法。我们的模型侧重于 fMRI 活动与多种模态之间的关系，即从查看的图像中提取的视觉特征和从查看的图像类别中获得的语​​义特征。此外，为了准确估计未用于训练的图像类别，除了训练数据的图像类别之外，我们的半监督框架还结合了从其他图像类别获得的视觉和语义特征。所提出模型的估计性能优于脑解码领域现有的最先进模型，识别准确率超过 95%。结果还表明，当训练样本数量较少时，加入额外的图像类别信息非常有效。我们的半监督框架对于大脑活动模式不足但视觉刺激足够的大脑解码领域具有重要意义。所提出模型的估计性能优于脑解码领域现有的最先进模型，并实现了超过 95% 的识别准确率。结果还表明，当训练样本数量较少时，加入额外的图像类别信息非常有效。我们的半监督框架对于大脑活动模式不足但视觉刺激足够的大脑解码领域具有重要意义。所提出模型的估计性能优于脑解码领域现有的最先进模型，并实现了超过 95% 的识别准确率。结果还表明，当训练样本数量较少时，加入额外的图像类别信息非常有效。我们的半监督框架对于大脑活动模式不足但视觉刺激足够的大脑解码领域具有重要意义。