Liver tumor segmentation using 2.5D UV-Net with multi-scale convolution

Highlights

• A novel 2.5D UV-Net is proposed to balance the memory consumption and 3D context.

• Multi-scale convolution structure is further fused into UV-Net, constructing the UV-Net-Multi-scale, which realizes multi-scale feature extraction with identical computing resources.

• An efficient preprocessing method of removing mean energy is introduced to reduce the difference between CT images to ensure the feature consistency of different patients.

Abstract

Liver tumor segmentation networks are generally based on U-shaped encoder-decoder network with 2D or 3D structure. However, 2D networks lose the inter-layer information of continuous slices and 3D networks might introduce unacceptable parameters for GPU memory. As a result, 2.5D networks were proposed to balance the memory consumption and 3D context. Different from the canonical 2.5D design, which utilizes a 2D network combined with RNN, we propose a new 2.5D design called UV-Net to encode the inter-layer information in the context of 3D convolution, and reconstruct the high-resolution results with 2D deconvolution. At the same time, the multi-scale convolution structure enables multi-scale feature extraction without extra computational cost, which effectively mines structured information, reduces information redundancy, strengthens independent features, and makes feature dimension sparse, to enhance network capacity and efficiency. Combined with the proposed preprocessing method of removing mean energy, UV-Net significantly outperforms the existing methods in liver tumor segmentation and especially improves the segmentation accuracy of small objects on the LiTS2017 dataset.

Introduction

Liver cancer is a serious threat to human health, the incidence rate of liver cancer ranking the sixth, and the mortality rate ranking the fourth among all cancers [1]. Modern medicine urgently demands of an efficient and accurate diagnostic method for liver lesion resection. Recent advances in computer vision have accelerated the development of some classical image processing tasks(e.g., image classification [62,63], object detection [64,65] and image segmentation [66,67]). Consequently, liver tumor segmentation has been aided with more and more advanced segmentation algorithms.

The raw data in biomedical image segmentation task is usually 3D(Fig. 1) data containing correlation between slices, which is the inter-layer information. Because of the unique encoder-decoder structure and the innovation of skip-connection structure, the segmentation ability of U-shaped network has been greatly improved, therefore many medical image segmentation networks are built on the basis of U-shaped network. Ordinary 2D network(based on 2D U-Net [2]) is trained by slices, making full use of the two-dimensional plane spatial information in each slice, however ignoring the information between slices leads to the decrease of segmentation accuracy. 3D network (based on V-Net [3]) utilizes intra-layer and inter-layer information to extract features through 3D convolution, which learns 3D context information. However, the large amount of memory consumption and computational burden in 3D networks is a great challenge for network training. Herein, many 2.5D segmentation networks have been proposed to balance the memory consumption and 3D learning capability. Recurrent neural networks, especially LSTM(i.e., Long Short Term Memory) [4], an effective model to process sequential data [5,6], could extract context information of 3D CT images by RNN(i.e., Recurrent Neural Network) structure, referring to Refs. [[7], [8], [9]]. However, the traditional 2.5D methods are designed as a 2D network with the RNN network. Actually, the methods based on RNN is not the optimal choice. On the one hand, the RNN structure is not suitable for parallel training on GPU, different from CNN(i.e., Convolutional Neural Network). On the other hand, when continuous slices are regarded as sequential data, the network structure of CNN + RNN cannot effectively solve the long-term dependence problem between continuous slices, consequently reducing the accuracy and the training efficiency. Therefore, we did not choose the 2.5D network with CNN + RNN structure, while choosing to combine 2D and 3D networks instead to construct a 2.5D network with a new architecture.

In this paper, we propose a new 2.5D segmentation network, called UV-Net, which simultaneously integrates the 2D design(i.e., U-Net [2]) and 3D design(i.e., V-Net [3]). Briefly, a 3D encoder is used to capture the 3D spatial context while a 2D decoder is used to maintain the high in-plane resolution. The improved 2.5D network uses continuous multi-layer slices as inputs and only outputs one label prediction, using inter-layer information between adjacent slices for targeted prediction.

In LiTS2017 [54] dataset, the size of target livers and tumors varies greatly in slices across different patients, even for the same patient. In current 2.5D design, the feature extraction by convolution with fixed scale could get the receptive field with fixed scale, limiting the network's ability to capture the object when the scale of the object varies greatly, which contributes to the inferior segmentation performance. Moreover, the feature maps of each dimension of the same convolution have redundant information. Hence, we proposed the UV-Net-Multi-scale, which fuses the multi-scale features extraction into network, on the basis of UV-Net. The output after realizing multi-scale convolution is to extract features on multiple scales, which not only reduces information redundancy, but also strengthens the independence of features and realizes feature dimension sparseness [[10], [11], [12], [13], [14], [15]], eventually enhancing network fitting ability and accelerating convergence. To realize multi-scale network, we refer to the Inception [16] structure, which consists of multiple independent convolution paths, aiming to obtain multiple independent multi-scale information, consequently overcome the difficulties above.

Benefiting from our novel 2.5D structure, combined with multi-scale convolution and our preprocessing method of removing mean energy, UV-Net significantly outperforms the recent algorithms in liver tumor segmentation and especially improves the segmentation accuracy of small objects on the LiTS2017 [54] dataset.