Memory-efficient deep learning inference with incremental weight loading and data layout reorganization on edge systems

Abstract

Pattern recognition applications such as face recognition and agricultural product detection have drawn a rapid interest on Cyber–Physical–Social-Systems (CPSS). These CPSS applications rely on the deep neural networks (DNN) to conduct the image classification. However, traditional DNN inference models in the cloud could suffer from network delay fluctuations and privacy leakage problems. In this regard, current real-time CPSS applications are preferred to be deployed on edge-end embedded devices. Constrained by the computing power and memory limitations of edge devices, improving the memory management efficacy is the key to improving the quality of service for model inference. First, this study explored the incremental loading strategy of model weights for the model inference. Second, the memory space at runtime is optimized through data layout reorganization from the spatial dimension. In particular, the proposed schemes are orthogonal to existing models. Experimental results demonstrate that the proposed approach reduced the memory consumption by 61.05% without additional inference time overhead.

Introduction

In the era of Internet-of-Things (IoT), deploying deep neural networks (DNN) on Cyber–Physical–Social-Systems (CPSS) has drawn a rapid interest. Owning to the modern High-Performance-Computing-Communications (HPCC) techniques, deep learning applications have brought tremendous changes to the entire industry and research [1], [2], [3]. In the 2012 ILSVRC (ImageNet Large Scale Visual Recognition Challenge) competition, the deep neural network Alexnet [4] won the championship with a 17% error rate, and thereafter the won the championship. Their solutions deepen the network layers and increase the number of network parameters to obtain higher generalization and accuracy. Until 2015, the ResNet [5] model used a cross-layer connection to partially solve the problem of gradient explosion or disappearance during training, further reducing the error rate to 3.57%. Meanwhile, these approaches increased the training complexity and hardware overhead.

In recent years, a number of studies [6], [7], [8] are proposed to solve problems or optimize the performance in Cyber–Physical Systems. For CPSS applications, with the development of modern high-performance hardware (such as Graphics Processing Unit (GPU) and Tensor Processing Unit (TPU)) and the availability of large-scale datasets, the accuracy of image classification and target recognition are getting higher. Though model network structures and operators are increasingly improved for the reduced cost of model training, the model inference has gradually become one of the major bottlenecks in deep learning applications. Limited by network delay and implicit protection, models are preferred to be deployed in edge-end embedded devices independently. As the improvement of the edge device computing power, low-cost and general-purpose CPU is capable of conducting the deep learning network inference stage [9]. However, the challenge of deploying the model on edge devices is its huge DRAM requirement.

The main memory overhead of the CNN model [10] involves two parts: (1) Deployment Memory: After the CNN training, we need to deploy the model structures and load the model weights into the memory during inference. For example, the weights of Alexnet consume more than 200 MB DRAM space, which is a great burden for edge devices with limited resources. (2) Run-time Memory: During model inference, additional memory space is needed to store the feature maps generated by the network and the intermediate results of the parallel calculation of the operators. For the Deployment Memory, the network model structures and weights are often compressed for a smaller memory consumption [11], [12]. On the other hand, Low-rank Decomposition [13] can be used to reduce Run-Time Memory, but it does not take account of the memory management in the inference framework and thus cannot further optimize the memory allocation efficacy.

Many works have been proposed to enhance the main memory management efficacy for deep learning at the system or framework layer. Peng et al. [14] explored the memory swapping for GPU-based training by weighing the benefits of swapping and recalculation. Similarly, there are a lot of studies [15], [16] focusing on memory optimization by exchanging GPU cache and memory. Unfortunately, these works are only used for GPU-based training models, while the memory usage of the inference on edge devices is very different. Li [17] and Wu [18] enhanced the data layout to optimize the access performance of internal and external memory, but they paid little attention to the resource-constrained feature of edge devices, e.g., the insufficient memory during the inference phase.

This paper aims at optimizing the Deployment Memory and Run-time Memory of model inference on edge-end embedded devices at the same time. First, it considers the data usage granularity of different layers and reuses the weight memory space in the time dimension. Second, it considers the data usage of tensors, reorganizes the layout of the runtime Blob and Workspace tensors, and reduces the total tensor memory consumption in the spatial dimension. Experimental results with a wide range of workloads show that compared to the traditional inference framework, the proposed technique can reduce the memory consumption by as much as 61.05% without any time overhead.

This paper makes the following contributions:

• Identified that main memory could be occupied by tensors uselessly for a long time in the inference phase.

• Proposed an incremental weight loading scheme to aggressively reuse allocated memory for reduced memory consumption.

• Proposed a data layout reorganization scheme in runtime to avoid the fragmentation of small dispersed free memory spaces.

The rest of the paper is organized as follows. Section 2 introduces the background and the related work. Section 3 presents the motivation. Section 4 describes the design details of the proposed schemes. Section 5 presents the evaluation results with some discussions. Section 6 concludes this paper.