Elsevier

Medical Image Analysis

Volume 81, October 2022, 102537
Medical Image Analysis

Conditional GANs based system for fibrosis detection and quantification in Hematoxylin and Eosin whole slide images

https://doi.org/10.1016/j.media.2022.102537Get rights and content

Highlights

  • Fibrosis can be detected and quantified in Hematoxylin and Eosin stain (HE) slides by producing virtual Masson’s Trichrome stain (MT).

  • Conditional generative adversarial networks (cGAN) can learn texture-based transformation from HE to MT with improved image similarity metrics.

  • The proposed system can enable the assessment of both HE and MT on the same tissue slide, i.e. at the same cross section.

  • Segmenting fibrosis in virtual MT can achieve enhanced semantic segmentation metrics compared to direct fibrosis segmentation in HE stain.

  • The proposed system can help in enhancing the efficiencies in histology laboratories by reducing labor time and cost.

Abstract

Assessing the degree of liver fibrosis is fundamental for the management of patients with chronic liver disease, in liver transplants procedures, and in general liver disease research. The fibrosis stage is best assessed by histopathologic evaluation, and Masson’s Trichrome stain (MT) is the stain of choice for this task in many laboratories around the world. However, the most used stain in histopathology is Hematoxylin Eosin (HE) which is cheaper, has a faster turn-around time and is the primary stain routinely used for evaluation of liver specimens. In this paper, we propose a novel digital pathology system that accurately detects and quantifies the footprint of fibrous tissue in HE whole slide images (WSI). The proposed system produces virtual MT images from HE using a deep learning model that learns deep texture patterns associated with collagen fibers. The training pipeline is based on conditional generative adversarial networks (cGAN), which can achieve accurate pixel-level transformation. Our comprehensive training pipeline features an automatic WSI registration algorithm, which qualifies the HE/MT training slides for the cGAN model. Using liver specimens collected during liver transplantation procedures, we conducted a range of experiments to evaluate the detected footprint of selected anatomical features. Our evaluation includes both image similarity and semantic segmentation metrics. The proposed system achieved enhanced results in the experiments with significant improvement over the state-of-the-art CycleGAN learning style, and over direct prediction of fibrosis in HE without having the virtual MT step.

Introduction

Determining the degree of liver fibrosis is a fundamental task in the management of chronic liver disease (CLD), in liver transplants procedures, and liver disease research. CLD is major global public health problem with significant mortality, morbidity, and negative economic impact. Sustained and prolonged liver injury that occurs in CLD causes fibrosis and as fibrosis progresses, increasing liver dysfunction ensues. While there are non-invasive techniques to assess the degree of liver fibrosis, some clinical scenarios require histopathologic evaluation of liver biopsy for a more robust diagnostic documentation. Histopathological evaluation of biopsy-acquired liver tissue, as illustrated in Fig. 1a, remains the gold standard methodology in clinical practice (Chalasani, Younossi, Lavine, Charlton, Cusi, Rinella, Harrison, Brunt, Sanyal, 2018, Eslam, Newsome, Anstee, Targher, Gomez, Zelber-Sagi, Wong, Dufour, Schattenberg, Arrese, et al., 2020, Eslam, George, 2020, Mak, Yuen, Seto, 2020). Typically, a liver biopsy specimen is sliced into 4-micron thin flat sheets that are mount on a glass slide. Next, the colorless tissues are dyed with certain stains. The Hematoxylin and Eosin (HE) is the most routinely used stain and it provides detailed morphological information of the cells with distinct visual separation of the nuclear and cytoplasmic structures (see Fig. 1b). However, it provides little visual contrast between the cellular structures and fibrous tissue. On the other hand, Masson’s Trichrome (MT) has a blue pigment that binds to normal or abnormal fibrous tissue (collagen I) to provide contrast with the cellular compartment, which is colored dark red (Krishna, 2013). Sample MT images are shown in Fig. 1c. Staining a single tissue slice using HE and MT simultaneously is impossible. Each stain must be applied on different tissue sections which show a slightly different distribution of microanatomy elements.

The emergence of whole-slide imaging technology drives the field of pathology informatics by enabling scanning tissue slides at high resolution and storing them digitally (Pallua et al., 2020). Along with the latest advances in deep-learning and computer vision, virtual copies of the digital slides can be produced by virtual staining” processes. Each copy is intended to represent the tissue appearance under certain stain. Previous studies attempted to use digital scans of colorless tissue (grayscale autofluorescence slides) to build virtual staining systems (Rivenson, de Haan, Wallace, Ozcan, 2020, Zhang, de Haan, Rivenson, Li, Delis, Ozcan, 2020). However, those systems face limitations as the used autofluorescence slides represents only the structural appearance of the tissue due to lack of histochemical labeling.

To the best of our knowledge, there are limited literature work that study direct transformation from HE to MT, in liver histopathology in particular. In Bayramoglu et al. (2017), a conditional GANs based model was developed and trained to transform from Hyperspectral Imaging to Trichrome. In Li et al. (2020), unlabeled tissue paired with H&E stained images were used in the model training targeting virtual H&E staining from unstained images. In this research work by Li et al. (2020), a conditional GAN based model that uses slides of resolution 1079 × 1079 pixels were used to estimate the intima thickness and area, and media area of the carotid artery using color-less unstained tissue. In Levy et al. (2020) study on nonalcoholic steatohepatitis liver biopsies, CycleGAN was used to transform WSI from HE to MT. Virtual MT WSIs were produced and assessed by pathologists, and high correlation was reported between staging of real and virtual stains. Generally, the generator in GANs architecture learns the model and tries to generate virtual samples. While the discriminator is responsible for differentiating between real and virtual samples. CycleGAN has two generators and two discriminators that are placed in a cyclic architecture (Zhu et al., 2017), unlike conditional generative adversarial networks (cGAN) that has one generator and one discriminator (see Figs. 2a and 4a). In de Haan et al. (2021), a transfer-learning style method was proposed that use load generators’ with the weights from generators that are trained on transforming from autofluerescence images to MT as per presented earlier in Zhang et al. (2020); de Haan et al. (2021). cGAN requires having pairs of real/virtual samples. Meanwhile in CycleGAN, each discriminator is trained to differentiate between a real sample with the virtual version of the same samples generated after passing through the two generators back and forth. CycleGAN, see Fig. 6, is a state-of-the-art method in terms of image transformation, because it has a superior capability in different applications where we cannot have paired training images, such as colorization and augmentation (Zhu, Park, Isola, Efros, 2017, Bashkirova, Usman, Saenko). We hypothesize that, if we have a paired HE-MT dataset, cGAN may be able to achieve more precise pixel-level estimation of MT appearance. To validate our hypothesis, we collected a set histopathological slides, and they were scanned to digital slides at 400× magnification ratio. Based on the results presented in Levy et al. (2020), CycleGAN was successful in obtaining good virtualization of MT stain on 200× magnification ratio, in spite of the unpaired training set. In this paper, we investigate how we can adapt the training data to be used in cGAN. We also compare the performance of using cGAN using the presented methods over CycleGAN. We support our study by having higher resolution at 400× magnification ratio. We also conduct a group of experiments to illustrate the ability of the proposed system to generate realistic MT stain and to perform fibrosis detection and quantification in the produced MT WSIs, compared to CycleGAN learning style.

To partially overcome the limitations of the previous HE-to-MT transformation attempts, we propose a novel digital pathology system (see Fig. 2a for the training framework, and Fig. 2b for the real-time usage framework) that, first, produces virtual MT from HE slides and, second, automatically detects the footprint of fibrous tissues in the produced virtual MT. The proposed system features a comprehensive training pipeline (Fig. 2a) that includes a process for tissue MT restaining, and an automatic rigid-body feature-based WSI automatic registration algorithm. The registration algorithm is incorporated in our pipeline to ensure near-perfect alignment of training pairs. The transformation model cGAN can learn accurate pixel-level model from paired HE and MT training images at the same tissue section. The proposed system can improve the accuracy of fibrosis staging by enabling examination of each tissue sections under MT and HE simultaneously, unlike using different sections in the current protocol. The proposed system improves the efficiency of fibrosis staging by eliminating the time and cost required to prepare physical MT slides. Details of our pipeline are fully described in the next section.

Section snippets

Methods

The proposed In-Silico fibrosis detection and quantification system has four main components: (1) WSI Rigid-Body Registration component that produces the pixel-level paired HE-MT images required for Pix2Pix training, (2) Color Normalization component that augments the digital histology images and to normalize the color appearance variability between slides, (3) Domain Transformation component that uses the input HE images to produce the virtual MT images, (4) Fibrosis Detection and

Material and data collection

The material used in the experiments consist of liver tissue specimens collected from 5 human subjects during liver transplantation surgeries. The study obtained the required IRB approvals and follows research ethics protocols. We also collected two additional sets of 16 and 5 pairs of slides, from 16 and 5 human subjects to be used as validation sets. The validation sets is collected from independent subjects and were converted to digital slides in independent setups and using different WSI

Experimental results

We start with presenting the performance of the registration algorithm, and qualitative and quantitative performance of the proposed stain model. We then move to the results of the ablation study that address the different dimensions of the proposed system by altering normalization technique, patch size, and magnification ratio are given. This is followed by qualitative and quantitative comparison between the proposed system and CycleGAN (Levy et al., 2020). Finally, we present the capability

Discussion

In this paper, we investigate the applicability to use cGAN model to perform transformation from HE to virtual MT. Next we studied whether the produced virtual MT can be used to segment fibrous tissue segments as a step towards fibrosis staging in chronic liver disease. We performed a group of experiments in order to explore the dimensions of this problem and to evaluate the performance of the proposed system qualitatively and quantitatively.

The proposed system encapsulates a WSI rigid-body

Conclusion and future work

In this paper, we propose a novel digital pathology system that can detect and quantify fibrosis in HE WSI. The proposed system is trained using a comprehensive pipeline that includes cGAN based HE-to-MT transformation model, WSI registration algorithm, and color-based detection model. We compared the proposed system with the different approaches at which fibrosis can be segmented Using U-net based model either directly from HE, or from the produced virtual MT. According to the set of

Declaration of Competing Interest

Authors declare that they have no conflict of interest.

Acknowledgments

The research project was reviewed and approved by the institutional research board at University of Louisville under IRB no. 21.0753. Also, this research work was supported in part by the U.S. National Science Foundation (NSF) under grant CNS1828521. Some of the systems and methods presented in this research work were filed as part of U.S. Patent Application 63/213,628.

References (31)

  • L. Bottou

    Stochastic gradient descent tricks

    Neural Networks: Tricks of the Trade

    (2012)
  • N. Chalasani et al.

    The diagnosis and management of nonalcoholic fatty liver disease: practice guidance from the american association for the study of liver diseases

    Hepatology

    (2018)
  • J. Duchi et al.

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Mach. Learn. Res.

    (2011)
  • M.A. Fischler et al.

    Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography

    Commun. ACM

    (1981)
  • K. de Haan et al.

    Deep learning-based transformation of H&E stained tissues into special stains

    Nat. Commun.

    (2021)
  • Cited by (5)

    View full text