Task relevance driven adversarial learning for simultaneous detection, size grading, and quantification of hepatocellular carcinoma via integrating multi-modality MRI
Graphical abstract
Introduction
Hepatocellular Carcinoma (HCC), as the second most frequent cause of cancer-related death globally (Liver et al., 2018), the detection, size grading, and multi-index quantification (i.e. the center point coordinates, max-diameter, and area) of HCC are clinically significant tasks for clinicians to assess and diagnose tumors (Goh et al., 2016, Liu, Yang, Chen, Wang, Huang, Zeng, Lin, Zeng, Guo, Zhou, et al., 2020, Zhao, Li, Kassam, Howey, Chong, Chen, Li, 2020). According to Wu et al. (2018) and Lu et al. (2011), HCC size at diagnosis is an independent prognostic factor for overall survival in HCC. The work Usta and Kayaalp (2020) proved that there is a significant association between maximum tumor diameter and microvascular invasion. And the work Gonzalez-Guindalini et al. (2013) indicated that the quantitative imaging allows robust evaluation of HCC response. As shown in Fig. 1, in the clinic, the detection, size grading, and multi-index quantification of HCC are manually performed by clinicians. It still has the disadvantages of manpower-consuming, time-consuming, subjective, and changeable (Lee et al., 2013). And it also suffers from the intra- and inter-observer variability (Kim et al., 2016). Thus, as shown in Fig. 1, if the detection, size grading, and multi-index quantification of HCC can be achieved simultaneously and automatically, it will greatly promote the clinical assessment and diagnosis of HCC.
Recently, an increasing number of works have been devoted to automatic tumor detection, size grading, or multi-index quantification. For instance, the work (Zhao et al., 2020b) attempted to use the T1FS for liver tumor detection. But it is limited to the use of single modality MRI, which ignores the potential complementary information among multi-modality MRI (Zhao et al., 2021). Although the work (Zhao et al., 2021) attempted to integrate multi-modality MRI for liver tumors diagnosis via using adversarial learning. The fusion and selection of multi-modality MRI are on the basis of the CNN module. Admittedly, the CNN-based module has shown great power in non-local feature extraction. But it suffers from ignoring the relevance of global information (Wang, Girshick, Gupta, He, 2018, Jaderberg, Simonyan, Zisserman, et al., 2015). Thus, using the CNN-based module for multi-modality fusion is incapable of capturing the relevance among multi-modality MRI. In work Ge et al. (2019) and Ruan et al. (2020), they are all focused on the simultaneous segmentation and multi-index quantification of medical images (i.e. kidney tumor and cardiac left ventricle). But these works can not be applied to HCC subjects well when lesion information is blurry or even invisible in single modality MRI because these methods lack an effective mechanism for multi-modality MRI fusion. To the best of our knowledge, no work attempted the automatic HCC detection, size grading, or multi-index quantification via integrating multi-modality MRI. In the clinic, integrating the information of in-phase, out-phase, T2-weighted imaging, and DWI sequences for HCC diagnosis have achieved success (Wu et al., 2019). It provides the clinical basis for using these four types of sequences for HCC diagnosis. But this work still depends on the manual delineation of the contour of the lesion area by clinicians.
It is still challenging to fuse and select multi-modality MRI for HCC detection, size grading, and multi-index quantification: the information of HCC on multi-modality MRI is diverse among different subjects, which leads to the fusion and selection of multi-modality MRI difficult. Specifically, as shown in Fig. 2, for subject1, there is relatively clear HCC information in T2FS modality. For subject2, there is relatively clear HCC information in DWI modality. For subject3, there is relatively clear HCC information on out-phase and DWI modalities. Moreover, for HCC patients, the information of HCC in some modalities are blurry or even invisible. This also reflects the importance of the mechanism of multi-modality fusion and selection. Overall, these existing works are incapable of simultaneous HCC detection, size grading, and multi-index quantification on multi-modality MRI due to the following limitations: (1) the lack of an effective mechanism to capture the relevance among multi-modality MRI information for multi-modality feature fusion and selection; (2) the lack of effective mechanism and constraint strategy to achieve mutual promotion of multi-task (i.e. detection, size grading, and quantification).
In this paper, we proposed a novel task relevance driven adversarial learning framework (TrdAL) for simultaneous HCC detection, size grading, and multi-index quantification. Our basic assumption is that capturing the relevance among multi-modality MRI of in-phase, out-phase, T2FS, and DWI can facilitate the feature fusion and enforcing the relationship among multi-task can refine the mutual promotion. And then, it will improve the performance of HCC detection, size grading, and multi-index quantification. Specifically, the TrdAL first obtains expressive feature of dimension reduction via four parallel CNN-base encoders, which greatly reduces the calculation of the model parameters when performing the feature fusion via using the modality-aware Transformer (MaTrans). Secondly, the proposed MaTrans is utilized for multi-modality MRI features fusion and selection, which solves the challenge of multi-modality information diversity via capturing the relevance among multi-modality MRI. Then, the innovative task relevance driven and radiomics guided discriminator (Trd-Rg-D) is used for adversarial learning. The Trd-Rg-D captures the internal high-order relationships to refine the performance of multi-task simultaneously, which provided an effective mechanism for multi-task prediction. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction. Lastly, a novel task interaction loss function is used for constraining the TrdAL, which enhances the mutual promotion between multi-task.
The main contributions of this work are as follows:
- •
For the first time, our proposed TrdAL method provides a time-saving, reliable, and stable tool, which achieves simultaneous HCC detection, size grading, and multi-index quantification via integrating multi-modality MRI of in-phase, out-phase, T2FS, and DWI.
- •
The proposed MaTrans encodes the position of multi-modality MRI to capture the relevance among multi-modality MRI, which refines the feature fusion and selection.
- •
The innovative Trd-Rg-D captures the internal high-order relationships among multi-task to refine the performance of multi-task simultaneously. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction.
- •
The TrdAL provides a constraint strategy of tasks interaction, which enforces the higher-order consistency among multi-task labels to achieve the united adversarial learning among multi-task of detection, size grading, and multi-index quantification.
Section snippets
Related work
Recently, clinical studies indicate that multi-modality MRI has important value in the diagnosis of HCC (Wu, Liu, Cui, Chen, Song, Xie, 2019, Park, Kim, Kim, Choi, Huh, Kim, Kim, Lee, 2020). However, no work attempted simultaneous HCC detection, size grading, and multi-index quantification on multi-modality MRI, and even a direct HCC multi-index quantification method is not available. With the excellent ability in feature extraction and nonlinear fitting of deep learning method, many works have
Methodology
As shown in Fig. 3, the proposed TrdAL integrates multi-modality MRI (i.e. in-phase, out-phase, T2FS, and DWI) for simultaneous HCC detection, size grading, and multi-index quantification. The TrdAL entirely works via four interdependent parts: (1) CNN-based feature extraction and dimension reduction (Section 3.1). It utilizes four parallel convolution paths to obtain the expressive feature of dimension reduction, which greatly reduces the calculation of the model parameters when performing the
Dataset
A labeled multi-modality MRI dataset is used to evaluate our TrdAL, which includes 10,800 slice images from 135 HCC subjects provided by the McGill University Health Centre. And each subject has corresponding multi-modality MRI of in-phase, out-phase, T2FS, and DWI. Therefore, for each modality, it contains 2700 slice images, which have health slice images and HCC slice images. According to the clinical criterion, the labels of multi-index quantification are obtained manually by using the
Results and analysis
The effectiveness of the proposed TrdAl is validated in the HCC detection, size grading, and multi-index quantification. Experimental results show that the TrdAL achieves excellent accuracy of HCC detection, size grading, and multi-index quantification. We conduct a set of experiments to evaluate the performance of our TrdAL, including: (1) performance comparison of HCC detection with state-of-the-art (SOTA) methods (Section 5.1.1); (2) ablation study of detection task (Section 5.1.2); (3)
Conclusions and discussion
For the first time, the proposed TrdAL achieves simultaneous HCC detection, size grading, and multi-index quantification via integrating multi-modality MRI. The proposed MaTrans encodes the position of multi-modality MRI to capture the relevance among multi-modality MRI, which refines feature fusion and selection. And then, the results processing integrates three tasks into a matrix, which makes it possible for multi-task interaction. Next, the innovative Trd-Rg-D captures the internal
Declaration of Competing Interest
None.
Acknowledgment
This work was funded by the China Scholarship Council (No. 202008370191).
References (46)
- et al.
Multi-label classification of multi-modality skin lesion via hyper-connected convolutional neural network
Pattern Recognit.
(2020) - et al.
Direct automated quantitative measurement of spine by cascade amplifier regression network with manifold regularization
Med. Image Anal.
(2019) - et al.
Mb-fsgan: joint segmentation and quantification of kidney tumor on ct by the multi-branch feature sharing generative adversarial network
Med. Image Anal.
(2020) - et al.
Segmentation and quantification of infarction without contrast agents via spatiotemporal generative adversarial learning
Med. Image Anal.
(2020) - et al.
User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability
Neuroimage
(2006) - et al.
Tripartite-GAN: synthesizing liver contrast-enhanced MRI to improve tumor detection
Med. Image Anal.
(2020) - et al.
United adversarial learning for liver tumor segmentation and detection of multi-modality non-contrast MRI
Med. Image Anal.
(2021) - et al.
Multi-indices quantification of optic nerve head in fundus image via multitask collaborative learning
Med. Image Anal.
(2020) - et al.
Multi-task learning for segmentation and classification of tumors in 3d automated breast ultrasound images
Med. Image Anal.
(2021) - et al.
Higher order conditional random fields in deep neural networks
European Conference on Computer Vision
(2016)
Transunet: Transformers make strong encoders for medical image segmentation
arXiv preprint arXiv:2102.04306
An image is worth 16x16 words: Transformers for image recognition at scale
arXiv preprint arXiv:2010.11929
Utnet: a hybrid transformer architecture for medical image segmentation
International Conference on Medical Image Computing and Computer-Assisted Intervention
K-Net: integrate left ventricle segmentation and direct quantification of paired echo sequence
IEEE Trans. Med. Imaging
Importance of tumor size as a prognostic factor after partial liver resection for solitary hepatocellular carcinoma: implications on the current AJCC staging system
J. Surg. Oncol.
Assessment of liver tumor response to therapy: role of quantitative imaging
Radiographics
Generative adversarial nets
Adv. Neural Inf. Process. Syst.
Mask R-CNN
Proceedings of the IEEE international conference on computer vision
Deep residual learning for image recognition
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Densely connected convolutional networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Spatial transformer networks
Adv. Neural Inf. Process. Syst.
Interobserver variability in gross tumor volume delineation for hepatocellular carcinoma
Strahlenther. Onkol.
Cited by (6)
BP-Net: Boundary and perfusion feature guided dual-modality ultrasound video analysis network for fibrous cap integrity assessment
2023, Computerized Medical Imaging and GraphicsEdge-Aware Multi-task Network for Integrating Quantification Segmentation and Uncertainty Prediction of Liver Tumor on Multi-modality Non-contrast MRI
2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)