Research explaining the behavior of convolutional neural networks (CNNs) has gained a lot of attention over the past few years. Although many visualization methods have been proposed to explain network predictions, most fail to provide clear correlations between the target output and the features extracted by convolutional layers. In this work, we define a concept, i.e., class-discriminative feature groups, to specify features that are extracted by groups of convolutional kernels correlated with a particular image class. We propose a detection method to detect class-discriminative feature groups and a visualization method to highlight image regions correlated with particular output and to interpret class-discriminative feature groups intuitively. The experiments showed that the proposed method can disentangle features based on image classes and shed light on what feature groups are extracted from which regions of the image. We also applied this method to visualize “lost” features in adversarial samples and features in an image containing a non-class object to demonstrate its ability to debug why the network failed or succeeded.

在过去的几年中，关于卷积神经网络（CNN）行为的研究得到了很多关注。尽管已提出了许多可视化方法来解释网络预测，但大多数方法未能在目标输出和卷积层提取的特征之间提供清晰的相关性。在这项工作中，我们定义了一个概念，即类别区分特征组，以指定由与特定图像类别相关的卷积核组提取的特征。我们提出了一种检测方法，以检测类别区分特征组，并提出一种可视化方法，以突出显示与特定输出相关的图像区域，并直观地解释类别区分特征组。实验表明，该方法可以根据图像类别分解特征，并阐明从哪些图像区域提取了哪些特征组。我们还应用了这种方法来可视化对抗样本中的“丢失”特征以及包含非类对象的图像中的特征，以展示其调试网络失败或成功原因的能力。