Convolutional neural networks (CNNs) have achieved superior accuracy in many visual-related tasks. However, the inference process through a CNN's intermediate layers is opaque, making it difficult to interpret such networks or develop trust in their operation. In this article, we introduce SIGN method for modeling the network's hidden layer activity using probabilistic models. The activity patterns in layers of interest are modeled as Gaussian mixture models, and transition probabilities between clusters in consecutive modeled layers are estimated to identify paths of inference. For fully connected networks, the entire layer activity is clustered, and the resulting model is a hidden Markov model. For convolutional layers, spatial columns of activity are clustered, and a maximum likelihood model is developed for mining an explanatory inference graph. The graph describes the hierarchy of activity clusters most relevant for network prediction. We show that such inference graphs are useful for understanding the general inference process of a class, as well as explaining the (correct or incorrect) decisions the network makes about specific images. In addition, SIGN provide interesting observations regarding hidden layer activity in general, including the concentration of memorization in a single middle layer in fully connected networks, and a highly local nature of column activities in the top CNN layers.

中文翻译：

---

SIGN：基于概率网络活动解释的统计推断图


卷积神经网络 (CNN) 在许多视觉相关任务中取得了卓越的准确性。然而，通过 CNN 中间层的推理过程是不透明的，因此很难解释此类网络或建立对其运行的信任。在本文中，我们介绍了使用概率模型对网络隐藏层活动进行建模的 SIGN 方法。感兴趣层中的活动模式被建模为高斯混合模型，并且估计连续建模层中的簇之间的转移概率以识别推理路径。对于全连接网络，整个层活动被聚类，所得模型是隐马尔可夫模型。对于卷积层，活动的空间列被聚类，并开发最大似然模型来挖掘解释性推理图。该图描述了与网络预测最相关的活动集群的层次结构。我们表明，这样的推理图对于理解类的一般推理过程以及解释网络对特定图像做出的（正确或不正确）决策很有用。此外，SIGN 还提供了有关隐藏层活动的有趣观察结果，包括全连接网络中单个中间层的记忆集中度，以及 CNN 顶层列活动的高度局部性。