Neural coding is one of the central questions in systems neuroscience for understanding how the brain processes stimulus from the environment, moreover, it is also a cornerstone for designing algorithms of brain-machine interface, where decoding incoming stimulus is highly demanded for better performance of physical devices. Traditionally researchers have focused on functional magnetic resonance imaging (fMRI) data as the neural signals of interest for decoding visual scenes. However, our visual perception operates in a fast time scale of millisecond in terms of an event termed neural spike. There are few studies of decoding by using spikes. Here we fulfill this aim by developing a novel decoding framework based on deep neural networks, named spike-image decoder (SID), for reconstructing natural visual scenes, including static images and dynamic videos, from experimentally recorded spikes of a population of retinal ganglion cells. The SID is an end-to-end decoder with one end as neural spikes and the other end as images, which can be trained directly such that visual scenes are reconstructed from spikes in a highly accurate fashion. Our SID also outperforms on the reconstruction of visual stimulus compared to existing fMRI decoding models. In addition, with the aid of a spike encoder, we show that SID can be generalized to arbitrary visual scenes by using the image datasets of MNIST, CIFAR10, and CIFAR100. Furthermore, with a pre-trained SID, one can decode any dynamic videos to achieve real-time encoding and decoding of visual scenes by spikes. Altogether, our results shed new light on neuromorphic computing for artificial visual systems, such as event-based visual cameras and visual neuroprostheses.

中文翻译：

---

使用深度神经网络从神经尖峰重建自然视觉场景。

神经编码是系统神经科学中了解大脑如何处理来自环境的刺激的核心问题之一，而且，它也是设计脑机接口算法的基石，在该机器中，对传入的刺激进行解码对于提高物理性能具有很高的要求设备。传统上，研究人员一直将功能磁共振成像（fMRI）数据作为感兴趣的神经信号解码视觉场景。但是，就称为神经尖峰的事件而言，我们的视觉感知以毫秒为单位的快速时间尺度运行。很少有研究使用尖峰解码。在这里，我们通过开发一种基于深度神经网络的新颖解码框架（称为尖峰图像解码器（SID））来实现此目标，该框架用于重建自然视觉场景，包括来自实验记录的视网膜神经节细胞群体的峰值的静态图像和动态视频。SID是一种端到端解码器，一端为神经尖峰，另一端为图像，可以直接对其进行训练，以便以高精度的方式从尖峰中重建视觉场景。与现有的fMRI解码模型相比，我们的SID在视觉刺激的重建上也表现出色。此外，借助尖峰编码器，我们证明可以通过使用MNIST，CIFAR10和CIFAR100的图像数据集将SID推广到任意视觉场景。此外，借助预训练的SID，可以解码任何动态视频，以实现尖峰对视觉场景的实时编码和解码。总之，我们的结果为人工视觉系统的神经形态计算提供了新的思路，