Deep learning techniques are increasingly being adopted for classification tasks over the past decade, yet explaining how deep learning architectures can achieve state-of-the-art performance is still an elusive goal. While all the training information is embedded deeply in a trained model, we still do not understand much about its performance by only analyzing the model. This paper examines the neuron activation patterns of deep learning-based classification models and explores whether the models' performances can be explained through neurons' activation behavior. We propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other examines the predictability of neurons' behavior using entropy. Our comprehensive experimental study reveals that both the community quality (modularity) and entropy are closely related to the deep learning models' performances, thus paves a novel way of explaining deep learning models directly from the neurons' activation pattern.

中文翻译：

---

关于深度学习架构中神经元社区的演变

在过去的十年中，深度学习技术越来越多地被用于分类任务，但解释深度学习架构如何实现最先进的性能仍然是一个难以实现的目标。虽然所有的训练信息都深深地嵌入在一个训练好的模型中，但仅通过分析模型，我们仍然对其性能了解不多。本文研究了基于深度学习的分类模型的神经元激活模式，并探讨了模型的性能是否可以通过神经元的激活行为来解释。我们提出了两种方法：一种将神经元的激活行为建模为图形并检查神经元是否形成有意义的社区，另一种方法使用熵检查神经元行为的可预测性。