A recent paper [1] claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to employ a representation derived from this processing to construct a novel object classifier. That paper, together with a series of subsequent papers [2] , [3] , [4] , [5] , [6] , [7] , [8] , claims to achieve successful results on a wide variety of computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they employ, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that are known to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to classifying arbitrary temporal artifacts of the data instead of stimulus-related activity. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the reported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit from the brain-derived representation. Together, our results illustrate the far-reaching implications of the temporal autocorrelations that exist in all neuroimaging data for classification experiments. Further, our results calibrate the underlying difficulty of the tasks involved and caution against overly optimistic, but incorrect, claims to the contrary.

中文翻译：

---

脑电图分类实验的块设计的危险和陷阱


最近的一篇论文 [1] 声称对通过 EEG 测量的观看 ImageNet 刺激的受试者诱发的大脑处理进行分类，并采用从该处理中得出的表示来构建新颖的对象分类器。该论文以及一系列后续论文 [2]、[3]、[4]、[5]、[6]、[7]、[8] 声称在各种计算机上取得了成功的结果视觉任务，包括对象分类、迁移学习以及使用通过脑电图测量的大脑衍生表征来生成描绘人类感知和思维的图像。我们新颖的实验和分析表明，他们的结果很大程度上取决于他们采用的块设计，其中给定类别的所有刺激都呈现在一起，而快速事件设计则失败，其中不同类别的刺激随机混合。块设计导致基于块级时间相关性对任意大脑状态进行分类，这些时间相关性已知存在于所有脑电图数据中，而不是刺激相关的活动。由于测试集中的每个试验都与相应训练集中的许多试验来自同一块，因此它们的块设计导致对数据的任意时间伪影而不是与刺激相关的活动进行分类。这使得多篇发表的论文中对该数据进行的所有后续分析无效，并对所有报告的结果提出质疑。我们进一步表明，用随机密码本构建的新型对象分类器的性能与用从脑电图数据提取的表示构建的新型对象分类器一样好，甚至更好，这表明用从脑电图数据提取的表示构建的分类器的性能并不好。受益于大脑衍生的表征。 总之，我们的结果说明了分类实验的所有神经影像数据中存在的时间自相关的深远影响。此外，我们的结果校准了所涉及任务的潜在难度，并警告不要过度乐观但不正确的相反主张。