Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of visual perception and mental states. However, due to the limitations of sample size and the lack of an effective reconstruction model, accurate reconstruction of natural images is still a major challenge. The current, rapid development of deep learning models provides the possibility of overcoming these obstacles. Here, we propose a deep learning-based framework that includes a latent feature extractor, a latent feature decoder, and a natural image generator, to achieve the accurate reconstruction of natural images from brain activity. The latent feature extractor is used to extract the latent features of natural images. The latent feature decoder predicts the latent features of natural images based on the response signals from the higher visual cortex. The natural image generator is applied to generate reconstructed images from the predicted latent features of natural images and the response signals from the visual cortex. Quantitative and qualitative evaluations were conducted with test images. The results showed that the reconstructed image achieved comparable, accurate reproduction of the presented image in both high-level semantic category information and low-level pixel information. The framework we propose shows promise for decoding the brain activity.

基于功能磁共振成像的大脑解码最近已经能够识别视觉感知和精神状态。然而，由于样本量的限制以及缺乏有效的重建模型，自然图像的精确重建仍然是一个重大挑战。当前深度学习模型的快速发展为克服这些障碍提供了可能。在这里，我们提出了一种基于深度学习的框架，包括潜在特征提取器、潜在特征解码器和自然图像生成器，以实现从大脑活动中准确重建自然图像。潜在特征提取器用于提取自然图像的潜在特征。潜在特征解码器根据来自高级视觉皮层的响应信号来预测自然图像的潜在特征。自然图像生成器用于根据自然图像的预测潜在特征和来自视觉皮层的响应信号生成重建图像。使用测试图像进​​行定量和定性评估。结果表明，重建图像在高级语义类别信息和低级像素信息方面均实现了与当前图像可比较、准确的再现。我们提出的框架显示出解码大脑活动的希望。