GenExp: Multi-objective pruning for deep neural network based on genetic algorithm

Abstract

Unstructured deep neural network (DNN) pruning have been widely studied. However, previous schemes only focused upon compressing the model’s memory footprint, which had led to relatively low reduction ratio in computational workload. This study demonstrates that the main reason behind is the inconsistent distribution of memory footprint and workload of the DNN model among different layers. Based on this observation, we propose to map the network pruning flow as a multi-objective optimization problem and design an improved genetic algorithm, which can efficiently explore the whole pruning structure space with both pruning goals equally constrained, to find the suitable solution that strikes a judicious balance between the DNN’s model size and workload. Experiments show that the proposed scheme can achieve up to $34\%$ further reduction on the model’s computational workload compared to the state-of-the-art pruning scheme [11], [33] for ResNet50 on the ILSVRC-2012 dataset. We have also deployed the pruned ResNet50 models on a dedicated DNN accelerator, and the measured data have shown a considerable $6\times$ reduction in inference time compared to FPGA accelerator implementing dense CNN model quantized in INT8 format, and a $2.27\times$ improvement in power efficiency over 2080Ti GPU-based implementations, respectively.

Introduction

Deep neural networks (DNNs) have achieved remarkable results in the fields of computer vision, machine translation, speech recognition, etc. However, the high computational and storage costs pose great challenges for deployment of the DNN-based algorithms, especially when embedded setting with limited hardware resource is considered. To overcome this hurdle, various schemes, including model quantization (reducing arithmetic precision of the parameters) [10], [7], [20], low-rank factorization (factorizing the weight matrix into low-rank matrices) [34], knowledge distillation (distilling the knowledge learned from a complex network and pass it to a small network) [17], [43], and network pruning (trimming unimportant weights) [12], [26], [9], [21], have been proposed and studied to compress the model size and reduce the computational workload effectively.

Although various neural network pruning schemes have been widely studied in the literature [12], [13], [33], [4], current unstructured pruning methods still suffers from two problems: Firstly, from the perspective of hardware implementation, the runtime performance of the deployed DNN model depends not only on DNN model’s memory footprint but also on its computational workload [41], [40], [39]. However, it is often difficult to explicitly constrain both model footprint and computational workload reduction ratios at the same time in the model pruning flow. For instance, previous study [41] has shown that neural network model’s computational workload was not positively correlated with the number of the parameters. Therefore, for unstructured neural network pruning, a good DNN model size compression result can not guarantee the same effect in computational workload. As being compared in Table 1, studies of [27], [11] both compressed the number of parameters of the ResNet50 model by 5$\times$, however, the achieved computational workload reduction rates are very different. The study of [12] compressed the model size of AlexNet by 9$\times$, while the reduction ratio achieved in workload is only 2.4$\times$. The work of [9] has shown an impressive compression ratio up to 17.6$\times$ for the same network model, however, the computational workload is only saved by 2.9$\times$. Through a layer-by-layer analysis of the sparse model reported by previous studies, we have found that the pruning method that only focuses parameter compression often chooses to greedily delete the parameters with low computational workload, such as fully connected layers. Therefore, the actual performance boost gained from network pruning by using previous schemes is relatively small compared to the numbers in model size reduction.

Secondly, the layer-wise sparsity ratios are inherently tricky to set compared to other network training hyper parameters. Many previous works [12], [24] have shown that, in DNNs, some layers are sensitive to pruning, while others can be pruned significantly without degrading the model in accuracy. Consequently, it needs significant manual efforts to explore the pruning rate setting for each layer of the DNN model to balance the model size, workload, and accuracy, which is usually sub-optimal and time-consuming. Moreover, in recent years, with the continuous deepening of the DNN model layer, the hand-crafted model compression method [12], [9], [32] cannot achieve effective pruning. Previous works [19], [27], [5] used saliency-based formulas (including L1 norm and Taylor expansion) to measure the importance of weights and passively obtain the sparsity rate of each layer through global comparison. However, this type of method cannot explicitly constrain the computational workload of the sparse model. The latest study of [30] proposed a histogram-based search strategy to determine the pruning ratios at each bucket, however, the correlation between model search and fine-tuning results is not verified. In addition, the search process of this method limits the accuracy loss threshold and does not fully consider the accuracy recovery ability of fine-tuning, so the obtained pruning compression ratio is lower than most pruning works.

In this paper, we seek to develop a new DNN pruning approach that targets at optimal pruning results in both memory footprint and computational workload, and thus can significantly improve the runtime performance of the pruned model when deployed on dedicated hardware accelerators. Inspired by the recent studies of [24], [23], we introduce a genetic algorithm-based multi-objective DNN pruning flow that performs sparsity architecture exploration to automatically obtain the suitable sparsity ratio for each layer before pruning/fine-tuning the baseline model as illustrated by Fig. 1. We name our scheme as GenExp. The main contributions of this study are:

• We propose a genetic algorithm-based multi-objective DNN pruning scheme (GenExp), which targets improved inference performance over previous studies when deployed on dedicated hardware accelerators. GenExp maps DNN pruning as a neural network sparsity architecture space exploration problem and automatically finds high-quality solutions under both the constraints of memory footprint and computational workload without manual settings.

• We propose several improvements to the genetic algorithm including combined Gaussian initialization, progressive shrinking mutation, and fine-grained crossover. The improved genetic algorithm can achieve better solutions in the continuous sparsity architecture space.

• Unlike previous studies, which only report estimated performance numbers, we have verified the effectiveness of the proposed scheme on a dedicated DNN hardware accelerator [37]. Real-world experiment results show that GenExp can effectively reduce the ResNet50 model’s execution time by up to 6 $\times$ at the cost of 0.1% accuracy drop compared to the dense model for image classification task on ImageNet, and achieve 10$\times$ model size compression and 6.5$\times$ workload reduction of the YOLOv2 model for object detection task on PASCAL VOC dataset.

The remainder of the paper is organized as follows. In Section 2, we describe the related work. The details of our proposed GenExp method are presented in Section 3. Experimental comparisons with other state-of-the-art pruning methods and ablation study are provided in Section 4. We conclude our work and plan of further improvements in Section 5.