Despite the great success of deep convolutional neural networks (DCNNs) in computer vision tasks, their black-box aspect remains a critical concern. The interpretability of DCNN models has been attracting increasing attention. In this work, we propose a novel model, Feature-fLOW INterpretation (FLOWIN) model, to interpret a DCNN by its feature-flow. The FLOWIN can express deep-layer features as a sparse representation of shallow-layer features. Based on that, it distills the optimal feature-flow for the prediction of a given instance, starting from deep layers to shallow layers. Therefore, the FLOWIN can provide an instance-specific interpretation, which presents its feature-flow units and their interpretable meanings for its network decision. The FLOWIN can also give the quantitative interpretation in which the contribution of each flow unit in different layers is used to interpret the net decision. From the class-level view, we can further understand networks by studying feature-flows within and between classes. The FLOWIN not only provides the visualization of the feature-flow but also studies feature-flow quantitatively by investigating its density and similarity metrics. In our experiments, the FLOWIN is evaluated on different datasets and networks by quantitative and qualitative ways to show its interpretability.

中文翻译：

---

深度卷积神经网络的特征流解释

尽管深度卷积神经网络 (DCNN) 在计算机视觉任务中取得了巨大成功，但它们的黑盒方面仍然是一个关键问题。DCNN 模型的可解释性越来越受到关注。在这项工作中，我们提出了一种新颖的模型，即特征流解释 (FLOWIN) 模型，通过其特征流来解释 DCNN。FLOWIN 可以将深层特征表示为浅层特征的稀疏表示。在此基础上，它为给定实例的预测提取最佳特征流，从深层到浅层。因此，FLOWIN 可以提供特定于实例的解释，它为网络决策呈现其特征流单元及其可解释的含义。FLOWIN 还可以给出定量解释，其中使用不同层中每个流单元的贡献来解释净决策。从类级别的角度来看，我们可以通过研究类内和类之间的特征流来进一步理解网络。FLOWIN 不仅提供了特征流的可视化，而且还通过研究特征流的密度和相似性度量来定量研究特征流。在我们的实验中，FLOWIN 通过定量和定性的方式在不同的数据集和网络上进行评估，以显示其可解释性。FLOWIN 不仅提供了特征流的可视化，而且还通过研究特征流的密度和相似性度量来定量研究特征流。在我们的实验中，FLOWIN 通过定量和定性的方式在不同的数据集和网络上进行评估，以显示其可解释性。FLOWIN 不仅提供了特征流的可视化，而且还通过研究特征流的密度和相似性度量来定量研究特征流。在我们的实验中，FLOWIN 通过定量和定性的方式在不同的数据集和网络上进行评估，以显示其可解释性。