Abstract
To study the influence of different sequences of magnetic resonance imaging (MRI) images on the segmentation of hepatocellular carcinoma (HCC) lesions, the U-Net was improved. Moreover, deep fusion network (DFN), data enhancement strategy, and random data (RD) strategy were introduced, and a multisequence MRI image segmentation algorithm based on DFN was proposed. The segmentation experiments of single-sequence MRI image and multisequence MRI image were designed, and the segmentation result of single-sequence MRI image was compared with those of convolutional neural network (FCN) algorithm. In addition, RD experiment and single-input experiment were also designed. It was found that the sensitivity (0.595 ± 0.145) and DSC (0.587 ± 0.113) obtained by improved U-Net were significantly higher than the sensitivity (0.405 ± 0.098) and DSC (0.468 ± 0.115, ) obtained by U-Net. The sensitivity of multisequence MRI image segmentation algorithm based on DFN (0.779 ± 0.015) was significantly higher than that of FCN algorithm (0.604 ± 0.056, ). The multisequence MRI image segmentation algorithm based on the DFN had higher indicators for liver cancer lesions than those of the improved U-Net. When RD was added, it not only increased the DSC of the single-sequence network enhanced by the hepatocyte-specific magnetic resonance contrast agent (Gd-EOB-DTPA) by 1% but also increased the DSC of the multisequence MRI image segmentation algorithm based on DFN by 7.6%. In short, the improved U-Net can significantly improve the recognition rate of small lesions in liver cancer patients. The addition of RD strategy improved the segmentation indicators of liver cancer lesions of the DFN and can fuse image features of multiple sequences, thereby improving the accuracy of lesion segmentation.
1. Introduction
Hepatocellular carcinoma (HCC) is a common malignancy disease with a high incidence in the global cancer statistics. So far, the global incidence rate has steadily increased to more than 672,000 people per year. In China, there are more than half of the patients, which seriously threatens the health of Chinese residents [1]. Clinical studies showed that accurate segmentation of HCC lesions from high-quality magnetic resonance imaging (MRI) is a very important link in the treatment process [2, 3]. However, traditional medical image segmentation segmented the lesions according to the shallow features of the images and relied on the clinical experience of the doctors. The addition of different subjective factors leads to different segmentation boundaries of lesions, resulting in large errors. In addition, the tedious and repetitive workload imposes a burden on medical staff [4, 5]. Therefore, the automatic and accurate identification and segmentation of HCC lesions are urgently needed to help doctors make diagnosis and improve efficiency. At present, HCC lesions are mostly segmented on CT images, while MRI images are rarely segmented [6, 7]. Bonanno et al. [8] firstly filtered the image and then used the watershed algorithm to obtain many small, segmented regions and merge them into several large tumor candidate regions. The final segmentation result was selected manually, but the effect was not good. Many relevant studies suggested that although HCC lesion segmentation is realized based on traditional machine learning algorithm, its high dependence on specified tasks and nonautomatic disadvantages cannot be ignored [9, 10].
With the development of artificial intelligence and information, deep convolutional neural network has been used in the segmentation of medical images, and its performance is far superior to traditional machine learning algorithms [11, 12]. Souhami et al. [13] proposed a 3D deep full convolutional neural network (FCNN) architecture to segment the lesions in MRI images of the prostate gland by combining the characteristics of the lesions and the information of its surrounding tissues. The results showed that the phase Dice coefficient was 0.878, which was much higher than the best result based on nondeep learning algorithm, and its speed was increased by about 350 times. However, its disadvantages cannot be ignored, for various reasons can result in inaccurate segmentation of lesions. As MRI images of different sequences reflect the information of the lesions from different angles, their complementary image information can reflect the features of the lesions in a comprehensive way [14]. Mitsala et al. [15] segmented brain tumors based on four MRI images and deep neural network. It was found that the complementation of multiple MRI image information can improve the segmentation performance. However, there are few reports on the focus segmentation of DCNN combined with multisequence MRI images. In this study, U-Net was improved. Deep fusion network, data enhancement strategy, and random data (RD) strategy were introduced, and a multisequence MRI image segmentation algorithm based on deep fusion network (DFN) was proposed. The segmentation experiments of single-sequence MRI image and multisequence MRI image were designed, and the segmentation result of single-sequence MRI image was compared with those of DCNN algorithm. In addition, RD experiment and single-input experiment were also designed. This study aimed to provide a theoretical reference for automatic segmentation of liver cancer lesions by multisequence MRI image segmentation algorithm based on DFN.
2. Methods
2.1. DCNN
The CNN model is inspired by biology and has excellent performance in medical image processing. It contains several basic modules. As the core of the CNN model, the convolutional layer occupies a crucial position. Its process is expressed as follows:
In equation (1), is the input, is the bias term on i, and the convolution kernel is . The commonly used activation function F of the activation layer is the Sigmoid function:
Equation (2) is derived as follows:
In equation (3), when the absolute value of the independent variable x becomes larger, the gradient of the function will tend to be flatter. ReLU function is as follows:
Equation (4) not only can greatly reduce the amount of calculation but also has sparseness. It mainly solves the problem of network performance degradation [16], so it is widely used in deep convolutional neural networks.
The pooling layer is a simple downsampling operation, which can reduce the amount of calculation and the amounts of parameters. Among them, average pooling and maximum pooling are two commonly used pooling layers. On the contrary, the depooling layer is an upsampling operation. If it is the maximum pooling during the network forwarding process, the position of the activation value is recorded; otherwise, the position of the nonmaximum value is set to 0. Maximum pooling (2 × 2) and depooling are shown in Figure 1.
The function of the random inactivation layer is preventing overfitting, which is defined as follows:
In equations (5) and (6), is the probability value, is the original output, is the output after random inactivation, and n is the number of layers. In this layer, the number within (0, 1) is set as the probability, and a binary distribution vector is generated during each training. When the result is 0, the output is 0; otherwise, it is the original number. In this way, it can avoid overweighting and prevent overfitting. In this article, Softmax is defined as follows:
In equation (7), is the number of target categories and is the input vector, whose cross-entropy loss is defined as follows:
In equation (8), it is supposed that is the final output result, is the coded label of the gold standard, and and represent the label of the gold standard and the category of the first point on the prediction result, respectively, whose results are only 0 or 1.
The U-Net was improved. As there are five pooling layers on the original U-Net network structure, the segmentation target may disappear during the calculation process when it is too small. Therefore, only two pooling layers are retained, and a convolution module composed of a convolution layer, a batch regularization layer [17], and an activation layer is added in front of each pooling layer, as shown in Figure 2.
The original image was input into the improved U-Net network. After two convolution modules, the image was compressed to a quarter of the original. Then, three convolutional layers were used to encode the extracted image features. Finally, after three deconvolution modules, the processed image was output. Some of the surface features in the convolution module were also connected to the deconvolution module, as shown in Figure 3.
2.2. Deep Integration and Data Enhancement Strategy
The algorithm proposed in this paper was added with deep fusion to integrate and increase the segmentation results of a single sequence of MRI images. The schematic diagram of dual-input deep fusion is shown in Figure 4.
It is supposed that two sequences of MRI images are input to two improved U-Net networks (U-Net 1 and U-Net 2), and a dual-input convolutional neural network is added after the two improved U-Net networks, which is expressed as follows:
In equation (9), and represent the input and output of the network i, respectively, and is the mapping relationships. The total loss function is as follows:
In equation (10), is the loss weight of the subnetwork, is the output of the fusion network, and is the deviation of the final result from the gold standard [18].
Data enhancement can solve the problem of small amount of image feature data. Due to the unfixed position of HCC lesions, random translation and mirroring plus random small-angle rotation were used to achieve various functions. Different patients have different tumor sizes and great differences, so the scaling ratio of 0.8 to 1.3 was randomly selected to obtain the multiscale input data [19].
2.3. Multisequence MRI Image Segmentation Algorithm Based on DFN
Based on the previous improvements to the U-Net network, deep fusion was introduced to deeply fuse the segmentation results of multisequence MRI images, and data enhancement strategies were adopted to construct a multisequence MRI image segmentation algorithm based on deep convolutional neural networks, as shown in Figure 5.
2.4. Experiments
In this research, 523 abdominal MRI images of 58 patients with liver cancer in X hospital were collected from July 4, 2018, to February 5, 2020. Siemens 3.0T magnetic resonance scanner was employed to obtain all MRI images. During the scanning, the patient maintained a supine position, and the parameters were set to conventional parameters. Two MRI images of hepatocyte-specific magnetic resonance contrast agent- (Gd-EOB-DTPA-) enhanced T1 sequence and portal phase (Portal) sequence were collected for each patient. The size of the input image was 256 × 256. Corresponding processing for MRI images of different sizes was made. (I) When it was less than 256, 0 was used to fill in. (II) When it was larger than 256 and smaller than 300, the image information was retained to the greatest extent by cropping the 256 × 256 size image. (III) When it was larger than 300, the image needed to be reduced to 256 × 256 size. Experienced radiology medical workers artificially delineated liver cancer lesions as the gold standard.
In the experiment, to solve the problem of network overfitting, the data enhancement strategy was adopted to increase the data to about 25,000.
For segmentation of single-sequence MRI images, the platform was Ubuntu 14.04 and the hardware was Processor E5-2650 v3 CPU and NVIDIA 1080 8G video memory GPU. A total of 15 Gd-EOB-DTPA-enhanced T1 sequence images of liver cancer patients were randomly selected for testing, and the performance of the improved U-Net and U-Net was compared.
Segmentation of multisequence MRI images was as follows. (I) For DFN segmentation, the platform was the same as above, and MRI images of one patient at a time were used for testing and the rest for training. (II) The RD strategy was adopted to compare the results with RD and without RD. The experimental framework is shown in Figure 6.
Single-input experiment was implemented to verify how the two sequences worked, and the experimental framework is shown in Figure 7.
2.5. Observation Indicators
In this paper, the evaluation indicators to measure the network model were sensitivity, precision, and comprehensive evaluation segmentation effect (DSC). For a prediction graph of N × H, the sensitivity is expressed as follows:
Precision is expressed as follows:
DSC is expressed as follows:
In equations (11)–(13), TP means the detected lesions are true positive, FP means the detected lesions are false positive, and FN means the detected lesions are false negative.
2.6. Statistical Methods
SPSS 22.0 was employed for data statistics and analysis. Mean ± standard deviation (͞x ± s) was how measurement data were expressed. The difference was statistically considerable with .
3. Results
3.1. Improved U-Net Performance Test
Figure 8 shows the comparison of segmentation indicators of the improved U-Net and U-Net. The sensitivity (0.786 ± 0.180) obtained by improved U-Net was significantly higher than the sensitivity (0.593 ± 0.062) obtained by U-Net, and the difference was significant (). The precision (0.759 ± 0.131) and DSC (0.732 ± 0.113) obtained by improved U-Net were both higher than the precision (0.686 ± 0.156) and DSC (0.663 ± 0.185) obtained by U-Net, but the differences were not significant ().
Figure 9 shows the comparison of segmentation indicators for small lesions of improved U-Net and U-Net. The sensitivity (0.595 ± 0.145) and DSC (0.587 ± 0.113) obtained by improved U-Net were significantly higher than the sensitivity (0.405 ± 0.098) and DSC (0.468 ± 0.115) obtained by U-Net, and the differences were significant (). The precision (0.601 ± 0.160) obtained by improved U-Net was slightly improved compared to the precision (0.520 ± 0.110) obtained by U-Net, but the difference was not significant (). It showed that the improved U-Net can significantly improve the recognition rate of small lesions in patients with liver cancer.
3.2. Comparison of Segmentation Effect of Improved U-Net and U-Net
Figure 10 shows the comparison of segmentation effects for large lesions of improved U-Net and U-Net. For obvious large lesions, the segmentation results of the improved U-Net and U-Net were close to the gold standard outlined by the medical staff, indicating that the segmentation effect of large lesions by the improved U-Net was not reduced but had a little improvement.
(a)
(b)
(c)
(d)
Figure 11 shows the comparison of segmentation effects for small lesions of improved U-Net and U-Net. While U-Net was recognizing and segmenting the small lesions, some nonlesion areas next to them were mistakenly regarded as lesions and were segmented. The reasons were that the fact that the target was too small and the characteristics of the lesion and surrounding tissues were regarded as a characteristic value, and there was no selection of characteristics in the middle, resulting in the current result. On the contrary, the improved U-Net was made up for this defect, but both the improved U-Net and U-Net lacked the ability to recognize extremely small lesions.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
3.3. Segmentation Results Based on DFN
Figure 12 shows the indicators based on the DFN segmentation algorithm. The average sensitivity was 0.779 ± 0.136, the average precision was 0.832 ± 0.141, the average DSC was 0.783 ± 0.075, and the median value of DSC was 0.869, which was relatively high.
Figure 13 shows the typical effect of segmentation based on DFN. Taking the gold standard as a reference, there was a dark tissue suspected of a lesion on the right side in Figure 13(a), which was only segmented from the Gd-EOB-DTPA-enhanced T1 sequence. In Figure 13(b), a part of the tissue was obviously missing, and the segmentation result of the portal vein sequence (portal) sequence in Figure 13(c) was even more missing. However, when the two sequences were fused, the segmentation result became obviously closer to the gold standard.
(a)
(b)
(c)
(d)
3.4. The Result after Addition of RD Strategy
Figure 14 shows the comparison of DSC based on DFN segmentation with and without RD. When RD was not added, the DSC (0.724 ± 0.103) of the Gd-EOB-DTPA sequence segmentation was very close to the DSC (0.726 ± 0.079) of the double sequence segmentation. The addition of the portal sequence had almost no effect on the segmentation results based on the DFN algorithm. When RD was added, it not only increased the DSC of the Gd-EOB-DTPA single-sequence network by 1% but also increased the DSC of the segmentation based on the DFN algorithm by 7.6%. It showed that the RD strategy can significantly improve the segmentation performance of liver cancer lesions based on the DFN algorithm.
Figure 15 shows the segmentation results based on the DFN with and without RD. When RD was not added, the segmentation results in Figures 15(C1) and 15(E1) were almost the same. Although the portal sequence segmentation result in Figure 15(D1) was significantly different from the Gd-EOB-DTPA-enhanced T1 sequence segmentation result in Figure 15(C1), it did not have any impact on the segmentation result in Figure 15(E1) based on the DFN algorithm, and it was far from the gold standard in Figure 15(B1). When RD was added, the segmentation results in Figures 15(C2) and 15(E2) were obviously different. The addition of portal sequence had a significant impact on the segmentation result in Figure 15(E2) based on the DFN algorithm, and it was close to the gold standard in Figure 15(B2).
3.5. Single-Input Experiment Results
Figure 16 shows the segmentation result of single input plus RD based on DFN. When only Gd-EOB-DTPA-enhanced T1 sequence images were input, the segmentation results in Figures 16(E1) and 16(C1) based on the DFN algorithm were almost the same, indicating that the segmentation did not rely on the portal sequence. When only the portal sequence images were input, the lesion area of Figure 16(C2) was not very obvious and was slightly larger than the gold standard. In Figure 16(D2), during segmentation of the lesion, there was a phenomenon that the blood vessel was mistakenly divided into the lesion, but the lesion area was similar to the gold standard. In Figure 16(E2), the response to the lesion and blood vessel was decreased, and it had an inhibitory effect on the segmented lesion area. It showed that the segmentation result of portal sequence was also suppressed by EOB sequence. The features of the two sequence MRI images inhibited each other, and the two were not responding at the same time. Only when two sequences were input at the same time, the algorithm based on DFN can get better results.
3.6. Overall Evaluation of Automatic Segmentation Methods
Figure 17 shows the comparison of the segmentation indicators of the four segmentation methods for liver cancer lesions. When improved U-Net segmentation was adopted, portal sequence’s recognition of liver cancer lesions was far worse than Gd-EOB-DTPA-enhanced T1 sequence, and the data of the former was lower than the data of the latter. The sensitivity of the DFN algorithm (0.779 ± 0.015) was significantly higher than that of the FCN + EOB algorithm (0.604 ± 0.056) (). The indicators of DFN for liver cancer lesions were higher than those of the improved U-Net, suggesting that the multisequence MRI image segmentation algorithm based on DFN had the best performance in the automatic segmentation of liver cancer lesions.
4. Discussion
The incidence of liver cancer has gradually increased in recent years, and the incidence of many areas with underdeveloped medical conditions in Africa and parts of Asia remains high [20]. Multisequence MRI plays a pivotal role in the treatment of liver cancer, and the features of different sequence images can be complementary. Since traditional manual segmentation of lesions has many shortcomings, there is an urgent need for methods to automatically and accurately identify and segment HCC lesions to help doctors diagnose and improve efficiency [21, 22]. Therefore, U-Net was improved and DFN and data enhancement strategies were introduced. The features of different sequences of MRI images were combined and RD strategy was adopted. A multisequence MRI image segmentation algorithm based on DFN was proposed. First, the performance of the network before and after the improvement was compared through a single-sequence MRI image segmentation experiment. It was found that the sensitivity obtained by improved U-Net (0.786 ± 0.180) was significantly higher than that obtained by U-Net (0.593 ± 0.062) (). For small lesions, the sensitivity (0.595 ± 0.145) and DSC (0.587 ± 0.113) obtained by improved U-Net were significantly higher than those obtained by U-Net (0.405 ± 0.098) and DSC (0.468 ± 0.115) (). It showed that the improved U-Net can significantly improve the recognition rate of small lesions in patients with liver cancer, and its precision for small lesions increased by about 0.8. However, both the improved U-Net and U-Net lacked the ability to recognize extremely small lesions, which was consistent with the results of O’Sullivan et al. [23]. Then, the residual unit was added to improve the structure of U-Net. As a result, the ability to recognize large lesions was significantly improved, while the ability to recognize very small lesions was still lacking.
The segmentation experiment of multisequence MRI image was implemented, and the results were compared with the segmentation results of single-sequence MRI image and DCCN algorithm. It was found that when the improved U-Net segmentation was performed, portal sequence was far less recognizable to liver cancer lesions than Gd-EOB-DTPA-enhanced T1 sequence, and the data of the former was lower than the data of the latter. The sensitivity of the DFN algorithm (0.779 ± 0.015) was significantly higher than that of the DCCN algorithm (0.604 ± 0.056) (). The indicators of DFN for liver cancer lesions were higher than those of the improved U-Net, indicating that the multisequence MRI image segmentation algorithm based on DFN had the best performance in the automatic segmentation of liver cancer lesions. In addition, a RD experiment was also designed. It was found that when RD was not added, the DSC (0.724 ± 0.103) of the Gd-EOB-DTPA sequence segmentation was very close to the DSC (0.726 ± 0.079) of the double sequence segmentation. When RD was added, it not only increased the DSC of the Gd-EOB-DTPA single-sequence network by 1% but also increased the DSC of segmentation based on the DFN algorithm by 7.6%. It meant that the RD strategy can significantly improve the segmentation performance of the algorithm proposed in this paper for liver cancer lesions. The single-input experiment revealed that when there was only one sequence of MRI images input, the subnetwork of the background image was input, and the output was the background image. In addition, the segmentation results of the portal sequence and the segmentation results of the EOB sequence inhibited each other. The features of the two sequence MRI images inhibited each other, and the two were not responding at the same time. Only when two sequences were input at the same time, the algorithm based on DFN can get better results. This coincided with the result of Rasouli et al. [24]. Then, head and neck organ segmentation on T1-weighted, T2-weighted, and T1-weighted fat-saturated images was performed, and then their features were merged. It was found that the segmentation result of a single image was conducive to improving the segmentation performance of the image.
5. Conclusion
U-Net was improved and DFN and data enhancement strategies were introduced. The features of different sequences of MRI images were combined and RD strategy was adopted. A multisequence MRI image segmentation algorithm based on DFN was proposed. First, the performance of the network before and after the improvement was compared through a single-sequence MRI image segmentation experiment. Secondly, a segmentation experiment of multisequence MRI images was performed and compared with the segmentation results of single-sequence MRI images and the DCCN algorithm. In addition, RD experiments and single-input experiments were also designed. The results showed that the improved U-Net can significantly improve the recognition rate of small lesions in patients with liver cancer. The addition of RD strategy improved the segmentation indicators of liver cancer lesions of the DFN and can fuse image features of multiple sequences, thereby improving the accuracy of lesion segmentation. The disadvantage of this paper is that the proposed algorithm is used for images with tumors and is used to train complete MRI images, and the algorithm is suitable for two-dimensional images. Later, the three-dimensional segmentation method will be explored to improve the segmentation performance.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no conflicts of interest.
Authors’ Contributions
Jinling Zhang and Jun Yang contributed equally to this work.