Background

Many scientific fields now use machine-learning tools to assist with complex classification tasks. In neuroscience, automatic classifiers may be useful to diagnose medical images, monitor electrophysiological signals, or decode perceptual and cognitive states from neural signals. However, such tools often remain black-boxes: they lack interpretability. A lack of interpretability has obvious ethical implications for clinical applications, but it also limits the usefulness of these tools to formulate new theoretical hypotheses.

New method

We propose a simple and versatile method to help characterize the information used by a classifier to perform its task. Specifically, noisy versions of training samples or, when the training set is unavailable, custom-generated noisy samples, are fed to the classifier. Multiplicative noise, so-called “bubbles”, or additive noise are applied to the input representation. Reverse correlation techniques are then adapted to extract either the discriminative information, defined as the parts of the input dataset that have the most weight in the classification decision, and represented information, which correspond to the input features most representative of each category.

Results

The method is illustrated for the classification of written numbers by a convolutional deep neural network; for the classification of speech versus music by a support vector machine; and for the classification of sleep stages from neurophysiological recordings by a random forest classifier. In all cases, the features extracted are readily interpretable.

Comparison with existing methods

Quantitative comparisons show that the present method can match state-of-the art interpretation methods for convolutional neural networks. Moreover, our method uses an intuitive and well-established framework in neuroscience, reverse correlation. It is also generic: it can be applied to any kind of classifier and any kind of input data.

Conclusions

We suggest that the method could provide an intuitive and versatile interface between neuroscientists and machine-learning tools.

中文翻译：

---

使用噪声、气泡和反向相关探索机器学习分类器

背景

许多科学领域现在使用机器学习工具来协助完成复杂的分类任务。在神经科学中，自动分类器可用于诊断医学图像、监测电生理信号或从神经信号解码感知和认知状态。然而，这些工具通常仍然是黑箱：它们缺乏可解释性。缺乏可解释性对临床应用具有明显的伦理意义，但它也限制了这些工具在制定新理论假设方面的有用性。

新方法

我们提出了一种简单而通用的方法来帮助表征分类器用于执行其任务的信息。具体来说，训练样本的噪声版本，或者当训练集不可用时，自定义生成的噪声样本被馈送到分类器。乘性噪声，即所谓的“气泡”或加性噪声应用于输入表示。然后，逆相关技术适用于提取判别信息（定义为输入数据集中在分类决策中具有最大权重的部分）和表示的信息（对应于最能代表每个类别的输入特征）。

结果

该方法用于通过卷积深度神经网络对书写数字进行分类；通过支持向量机对语音与音乐进行分类；以及通过随机森林分类器根据神经生理学记录对睡眠阶段进行分类。在所有情况下，提取的特征都很容易解释。

与现有方法的比较

定量比较表明，本方法可以匹配最先进的卷积神经网络解释方法。此外，我们的方法在神经科学中使用了一个直观且完善的框架，即反向相关。它也是通用的：它可以应用于任何类型的分类器和任何类型的输入数据。

结论

我们建议该方法可以在神经科学家和机器学习工具之间提供直观且通用的界面。