Elsevier

Medical Image Analysis

Volume 81, October 2022, 102554
Medical Image Analysis

Task relevance driven adversarial learning for simultaneous detection, size grading, and quantification of hepatocellular carcinoma via integrating multi-modality MRI

https://doi.org/10.1016/j.media.2022.102554Get rights and content

Highlights

  • For the first time, our proposed TrdAL method provides a time-saving, reliable, and stable tool, which achieves simultaneous HCC detection, size grading, and multiindex quantification via integrating multi-modality MRI of in-phase, out-phase, T2FS, and DWI.

  • The proposed MaTrans encodes the position of multi-modality MRI to capture the relevance among multi-modality MRI, which refines the feature fusion and selection.

  • The innovative Trd-Rg-D captures the internal high-order relationships among multitask to refine the performance of multi-task simultaneously. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction.

  • The TrdAL provides a constraint strategy of tasks interaction, which enforces the higher-order consistency among multi-task labels to achieve the united adversarial learning among multi-task of detection, size grading, and multi-index quantification.

Abstract

Hepatocellular Carcinoma (HCC) detection, size grading, and quantification (i.e. the center point coordinates, max-diameter, and area) by using multi-modality magnetic resonance imaging (MRI) are clinically significant tasks for HCC assessment and treatment. However, delivering the three tasks simultaneously is extremely challenging due to: (1) the lack of effective an mechanism to capture the relevance among multi-modality MRI information for multi-modality feature fusion and selection; (2) the lack of effective mechanism and constraint strategy to achieve mutual promotion of multi-task. In this paper, we proposed a task relevance driven adversarial learning framework (TrdAL) for simultaneous HCC detection, size grading, and multi-index quantification using multi-modality MRI (i.e. in-phase, out-phase, T2FS, and DWI). The TrdAL first obtains expressive feature of dimension reduction via using a CNN-based encoder. Secondly, the proposed modality-aware Transformer is utilized for multi-modality MRI features fusion and selection, which solves the challenge of multi-modality information diversity via capturing the relevance among multi-modality MRI. Then, the innovative task relevance driven and radiomics guided discriminator (Trd-Rg-D) is used for united adversarial learning. The Trd-Rg-D captures the internal high-order relationships to refine the performance of multi-task simultaneously. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction. Lastly, a novel task interaction loss function is used for constraining the TrdAL, which enforces the higher-order consistency among multi-task labels to enhance mutual promotion. The TrdAL is validated on a corresponding multi-modality MRI of 135 subjects. The experiments demonstrate that TrdAL achieves high accuracy of (1) HCC detection: specificity of 93.71%, sensitivity of 93.15%, accuracy of 93.33%, and IoU of 82.93%; (2) size grading: accuracy of large size, medium size, small size, tiny size, and healthy subject are 90.38%, 87.74%, 80.68%, 77.78%, and 96.87%; (3) multi-index quantification: the mean absolute error of center point, max-diameter, and area are 2.74mm, 3.17mm, and 144.51mm2. All of these results indicate that the proposed TrdAL provides an efficient, accurate, and reliable tool for HCC diagnosis in clinical.

Introduction

Hepatocellular Carcinoma (HCC), as the second most frequent cause of cancer-related death globally (Liver et al., 2018), the detection, size grading, and multi-index quantification (i.e. the center point coordinates, max-diameter, and area) of HCC are clinically significant tasks for clinicians to assess and diagnose tumors (Goh et al., 2016, Liu, Yang, Chen, Wang, Huang, Zeng, Lin, Zeng, Guo, Zhou, et al., 2020, Zhao, Li, Kassam, Howey, Chong, Chen, Li, 2020). According to Wu et al. (2018) and Lu et al. (2011), HCC size at diagnosis is an independent prognostic factor for overall survival in HCC. The work Usta and Kayaalp (2020) proved that there is a significant association between maximum tumor diameter and microvascular invasion. And the work Gonzalez-Guindalini et al. (2013) indicated that the quantitative imaging allows robust evaluation of HCC response. As shown in Fig. 1, in the clinic, the detection, size grading, and multi-index quantification of HCC are manually performed by clinicians. It still has the disadvantages of manpower-consuming, time-consuming, subjective, and changeable (Lee et al., 2013). And it also suffers from the intra- and inter-observer variability (Kim et al., 2016). Thus, as shown in Fig. 1, if the detection, size grading, and multi-index quantification of HCC can be achieved simultaneously and automatically, it will greatly promote the clinical assessment and diagnosis of HCC.

Recently, an increasing number of works have been devoted to automatic tumor detection, size grading, or multi-index quantification. For instance, the work (Zhao et al., 2020b) attempted to use the T1FS for liver tumor detection. But it is limited to the use of single modality MRI, which ignores the potential complementary information among multi-modality MRI (Zhao et al., 2021). Although the work (Zhao et al., 2021) attempted to integrate multi-modality MRI for liver tumors diagnosis via using adversarial learning. The fusion and selection of multi-modality MRI are on the basis of the CNN module. Admittedly, the CNN-based module has shown great power in non-local feature extraction. But it suffers from ignoring the relevance of global information (Wang, Girshick, Gupta, He, 2018, Jaderberg, Simonyan, Zisserman, et al., 2015). Thus, using the CNN-based module for multi-modality fusion is incapable of capturing the relevance among multi-modality MRI. In work Ge et al. (2019) and Ruan et al. (2020), they are all focused on the simultaneous segmentation and multi-index quantification of medical images (i.e. kidney tumor and cardiac left ventricle). But these works can not be applied to HCC subjects well when lesion information is blurry or even invisible in single modality MRI because these methods lack an effective mechanism for multi-modality MRI fusion. To the best of our knowledge, no work attempted the automatic HCC detection, size grading, or multi-index quantification via integrating multi-modality MRI. In the clinic, integrating the information of in-phase, out-phase, T2-weighted imaging, and DWI sequences for HCC diagnosis have achieved success (Wu et al., 2019). It provides the clinical basis for using these four types of sequences for HCC diagnosis. But this work still depends on the manual delineation of the contour of the lesion area by clinicians.

It is still challenging to fuse and select multi-modality MRI for HCC detection, size grading, and multi-index quantification: the information of HCC on multi-modality MRI is diverse among different subjects, which leads to the fusion and selection of multi-modality MRI difficult. Specifically, as shown in Fig. 2, for subject1, there is relatively clear HCC information in T2FS modality. For subject2, there is relatively clear HCC information in DWI modality. For subject3, there is relatively clear HCC information on out-phase and DWI modalities. Moreover, for HCC patients, the information of HCC in some modalities are blurry or even invisible. This also reflects the importance of the mechanism of multi-modality fusion and selection. Overall, these existing works are incapable of simultaneous HCC detection, size grading, and multi-index quantification on multi-modality MRI due to the following limitations: (1) the lack of an effective mechanism to capture the relevance among multi-modality MRI information for multi-modality feature fusion and selection; (2) the lack of effective mechanism and constraint strategy to achieve mutual promotion of multi-task (i.e. detection, size grading, and quantification).

In this paper, we proposed a novel task relevance driven adversarial learning framework (TrdAL) for simultaneous HCC detection, size grading, and multi-index quantification. Our basic assumption is that capturing the relevance among multi-modality MRI of in-phase, out-phase, T2FS, and DWI can facilitate the feature fusion and enforcing the relationship among multi-task can refine the mutual promotion. And then, it will improve the performance of HCC detection, size grading, and multi-index quantification. Specifically, the TrdAL first obtains expressive feature of dimension reduction via four parallel CNN-base encoders, which greatly reduces the calculation of the model parameters when performing the feature fusion via using the modality-aware Transformer (MaTrans). Secondly, the proposed MaTrans is utilized for multi-modality MRI features fusion and selection, which solves the challenge of multi-modality information diversity via capturing the relevance among multi-modality MRI. Then, the innovative task relevance driven and radiomics guided discriminator (Trd-Rg-D) is used for adversarial learning. The Trd-Rg-D captures the internal high-order relationships to refine the performance of multi-task simultaneously, which provided an effective mechanism for multi-task prediction. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction. Lastly, a novel task interaction loss function is used for constraining the TrdAL, which enhances the mutual promotion between multi-task.

The main contributions of this work are as follows:

  • For the first time, our proposed TrdAL method provides a time-saving, reliable, and stable tool, which achieves simultaneous HCC detection, size grading, and multi-index quantification via integrating multi-modality MRI of in-phase, out-phase, T2FS, and DWI.

  • The proposed MaTrans encodes the position of multi-modality MRI to capture the relevance among multi-modality MRI, which refines the feature fusion and selection.

  • The innovative Trd-Rg-D captures the internal high-order relationships among multi-task to refine the performance of multi-task simultaneously. Moreover, adding the radiomics feature as the prior knowledge into Trd-Rg-D enhances the detailed feature extraction.

  • The TrdAL provides a constraint strategy of tasks interaction, which enforces the higher-order consistency among multi-task labels to achieve the united adversarial learning among multi-task of detection, size grading, and multi-index quantification.

Section snippets

Related work

Recently, clinical studies indicate that multi-modality MRI has important value in the diagnosis of HCC (Wu, Liu, Cui, Chen, Song, Xie, 2019, Park, Kim, Kim, Choi, Huh, Kim, Kim, Lee, 2020). However, no work attempted simultaneous HCC detection, size grading, and multi-index quantification on multi-modality MRI, and even a direct HCC multi-index quantification method is not available. With the excellent ability in feature extraction and nonlinear fitting of deep learning method, many works have

Methodology

As shown in Fig. 3, the proposed TrdAL integrates multi-modality MRI (i.e. in-phase, out-phase, T2FS, and DWI) for simultaneous HCC detection, size grading, and multi-index quantification. The TrdAL entirely works via four interdependent parts: (1) CNN-based feature extraction and dimension reduction (Section 3.1). It utilizes four parallel convolution paths to obtain the expressive feature of dimension reduction, which greatly reduces the calculation of the model parameters when performing the

Dataset

A labeled multi-modality MRI dataset is used to evaluate our TrdAL, which includes 10,800 slice images from 135 HCC subjects provided by the McGill University Health Centre. And each subject has corresponding multi-modality MRI of in-phase, out-phase, T2FS, and DWI. Therefore, for each modality, it contains 2700 slice images, which have health slice images and HCC slice images. According to the clinical criterion, the labels of multi-index quantification are obtained manually by using the

Results and analysis

The effectiveness of the proposed TrdAl is validated in the HCC detection, size grading, and multi-index quantification. Experimental results show that the TrdAL achieves excellent accuracy of HCC detection, size grading, and multi-index quantification. We conduct a set of experiments to evaluate the performance of our TrdAL, including: (1) performance comparison of HCC detection with state-of-the-art (SOTA) methods (Section 5.1.1); (2) ablation study of detection task (Section 5.1.2); (3)

Conclusions and discussion

For the first time, the proposed TrdAL achieves simultaneous HCC detection, size grading, and multi-index quantification via integrating multi-modality MRI. The proposed MaTrans encodes the position of multi-modality MRI to capture the relevance among multi-modality MRI, which refines feature fusion and selection. And then, the results processing integrates three tasks into a matrix, which makes it possible for multi-task interaction. Next, the innovative Trd-Rg-D captures the internal

Declaration of Competing Interest

None.

Acknowledgment

This work was funded by the China Scholarship Council (No. 202008370191).

References (46)

  • J. Chen et al.

    Transunet: Transformers make strong encoders for medical image segmentation

    arXiv preprint arXiv:2102.04306

    (2021)
  • A. Dosovitskiy et al.

    An image is worth 16x16 words: Transformers for image recognition at scale

    arXiv preprint arXiv:2010.11929

    (2020)
  • Y. Gao et al.

    Utnet: a hybrid transformer architecture for medical image segmentation

    International Conference on Medical Image Computing and Computer-Assisted Intervention

    (2021)
  • R. Ge et al.

    K-Net: integrate left ventricle segmentation and direct quantification of paired echo sequence

    IEEE Trans. Med. Imaging

    (2019)
  • B.K. Goh et al.

    Importance of tumor size as a prognostic factor after partial liver resection for solitary hepatocellular carcinoma: implications on the current AJCC staging system

    J. Surg. Oncol.

    (2016)
  • F.D. Gonzalez-Guindalini et al.

    Assessment of liver tumor response to therapy: role of quantitative imaging

    Radiographics

    (2013)
  • I. Goodfellow et al.

    Generative adversarial nets

    Adv. Neural Inf. Process. Syst.

    (2014)
  • K. He et al.

    Mask R-CNN

    Proceedings of the IEEE international conference on computer vision

    (2017)
  • K. He et al.

    Deep residual learning for image recognition

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2016)
  • G. Huang et al.

    Densely connected convolutional networks

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2017)
  • M. Jaderberg et al.

    Spatial transformer networks

    Adv. Neural Inf. Process. Syst.

    (2015)
  • Y.S. Kim et al.

    Interobserver variability in gross tumor volume delineation for hepatocellular carcinoma

    Strahlenther. Onkol.

    (2016)
  • Kingma, D. P., Ba, J., 2014. Adam: a method for stochastic optimization. arXiv preprint...
  • Cited by (6)

    • Edge-Aware Multi-task Network for Integrating Quantification Segmentation and Uncertainty Prediction of Liver Tumor on Multi-modality Non-contrast MRI

      2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View full text