Information Bottleneck Theory on Convolutional Neural Networks

Abstract

Recent years, many researches attempt to open the black box of deep neural networks and propose a various of theories to understand it. Among them, information bottleneck (IB) theory claims that there are two distinct phases consisting of fitting phase and compression phase in the course of training. This statement attracts many attentions since its success in explaining the inner behavior of feedforward neural networks. In this paper, we employ IB theory to understand the dynamic behavior of convolutional neural networks (CNNs) and investigate how the fundamental features such as convolutional layer width, kernel size, network depth, pooling layers and multi-fully connected layer have impact on the performance of CNNs. In particular, through a series of experimental analysis on benchmark of MNIST and Fashion-MNIST, we demonstrate that the compression phase is not observed in all these cases. This shows us the CNNs have a rather complicated behavior than feedforward neural networks.

Appendix

In order to further verify our conclusions, we conduct additional experiments on the CIFAR-10 dataset [17]. This dataset consists of 60,000 $32\times 32$ colour images in 10 classes, with 6000 images per class. There are 50,000 training images and 10,000 test images.

In this experiment, the whole 50,000 training images and 10,000 test images are selected as our training dataset and test dataset respectively, which is the only different setting from Experiments and discussion section. Furthermore, because of the arithmetic of computing mutual information, we choose to average the image of three channels and turn it into a signal channel as input data.

Fig. 9

(Color figure online) MI path on CNNs with different convolutional layer widths on training data of CIFAR-10. The convolutional layer width of 3 networks are a 3-3-3-3-3-3, b 6-6-6-6-6-6, c 12-12-12-12-12-12

Fig. 10

(Color figure online) MI path on CNNs with different convolutional layer depths on training data of CIFAR-10. The depth of 4 networks are a $\hbox {depth}=2$, b $\hbox {depth}=4$, c $\hbox {depth}=7$, d $\hbox {depth}=10$. All these networks width are set to 6

Fig. 11

(Color figure online) MI path on CNNs with pooling layer on test data of CIFAR-10. a Without pooling layer, b with pooling layer. The width of convolutional layers are both 6-6-6 and the kernel size is fixed to $3\times 3$. Moreover, the MaxPooling2D layer is added after the layer 1 and the pooling size is set to $2\times 2$

The Figs. 9 and 10 show the MI with different widths and depths on training data respectively. Figure 11 shows the MI with pooling layer on test data. These results offer more proof about the IB theory.