A decade after speech was first decoded from human brain signals, accuracy and speed remain far below that of natural speech. Here we show how to decode the electrocorticogram with high accuracy and at natural-speech rates. Taking a cue from recent advances in machine translation, we train a recurrent neural network to encode each sentence-length sequence of neural activity into an abstract representation, and then to decode this representation, word by word, into an English sentence. For each participant, data consist of several spoken repeats of a set of 30–50 sentences, along with the contemporaneous signals from ~250 electrodes distributed over peri-Sylvian cortices. Average word error rates across a held-out repeat set are as low as 3%. Finally, we show how decoding with limited data can be improved with transfer learning, by training certain layers of the network under multiple participants’ data.

首次从人脑信号解码语音十年后，准确性和速度仍然远远低于自然语音。在这里，我们展示了如何以高精度和自然语速解码皮质电图。借鉴机器翻译的最新进展，我们训练了一个循环神经网络，将每个句子长度的神经活动序列编码为抽象表示，然后逐字解码该表示为英语句子。对于每个参与者，数据包括一组 30-50 个句子的多次口头重复，以及来自分布在侧裂皮层周围的约 250 个电极的同期信号。保留重复集的平均单词错误率低至 3%。最后，我们展示了如何通过在多个参与者的数据下训练网络的某些层，通过迁移学习来改进有限数据的解码。