Conditional GANs based system for fibrosis detection and quantification in Hematoxylin and Eosin whole slide images
Graphical abstract
Introduction
Determining the degree of liver fibrosis is a fundamental task in the management of chronic liver disease (CLD), in liver transplants procedures, and liver disease research. CLD is major global public health problem with significant mortality, morbidity, and negative economic impact. Sustained and prolonged liver injury that occurs in CLD causes fibrosis and as fibrosis progresses, increasing liver dysfunction ensues. While there are non-invasive techniques to assess the degree of liver fibrosis, some clinical scenarios require histopathologic evaluation of liver biopsy for a more robust diagnostic documentation. Histopathological evaluation of biopsy-acquired liver tissue, as illustrated in Fig. 1a, remains the gold standard methodology in clinical practice (Chalasani, Younossi, Lavine, Charlton, Cusi, Rinella, Harrison, Brunt, Sanyal, 2018, Eslam, Newsome, Anstee, Targher, Gomez, Zelber-Sagi, Wong, Dufour, Schattenberg, Arrese, et al., 2020, Eslam, George, 2020, Mak, Yuen, Seto, 2020). Typically, a liver biopsy specimen is sliced into 4-micron thin flat sheets that are mount on a glass slide. Next, the colorless tissues are dyed with certain stains. The Hematoxylin and Eosin (HE) is the most routinely used stain and it provides detailed morphological information of the cells with distinct visual separation of the nuclear and cytoplasmic structures (see Fig. 1b). However, it provides little visual contrast between the cellular structures and fibrous tissue. On the other hand, Masson’s Trichrome (MT) has a blue pigment that binds to normal or abnormal fibrous tissue (collagen I) to provide contrast with the cellular compartment, which is colored dark red (Krishna, 2013). Sample MT images are shown in Fig. 1c. Staining a single tissue slice using HE and MT simultaneously is impossible. Each stain must be applied on different tissue sections which show a slightly different distribution of microanatomy elements.
The emergence of whole-slide imaging technology drives the field of pathology informatics by enabling scanning tissue slides at high resolution and storing them digitally (Pallua et al., 2020). Along with the latest advances in deep-learning and computer vision, virtual copies of the digital slides can be produced by virtual staining” processes. Each copy is intended to represent the tissue appearance under certain stain. Previous studies attempted to use digital scans of colorless tissue (grayscale autofluorescence slides) to build virtual staining systems (Rivenson, de Haan, Wallace, Ozcan, 2020, Zhang, de Haan, Rivenson, Li, Delis, Ozcan, 2020). However, those systems face limitations as the used autofluorescence slides represents only the structural appearance of the tissue due to lack of histochemical labeling.
To the best of our knowledge, there are limited literature work that study direct transformation from HE to MT, in liver histopathology in particular. In Bayramoglu et al. (2017), a conditional GANs based model was developed and trained to transform from Hyperspectral Imaging to Trichrome. In Li et al. (2020), unlabeled tissue paired with H&E stained images were used in the model training targeting virtual H&E staining from unstained images. In this research work by Li et al. (2020), a conditional GAN based model that uses slides of resolution 1079 1079 pixels were used to estimate the intima thickness and area, and media area of the carotid artery using color-less unstained tissue. In Levy et al. (2020) study on nonalcoholic steatohepatitis liver biopsies, CycleGAN was used to transform WSI from HE to MT. Virtual MT WSIs were produced and assessed by pathologists, and high correlation was reported between staging of real and virtual stains. Generally, the generator in GANs architecture learns the model and tries to generate virtual samples. While the discriminator is responsible for differentiating between real and virtual samples. CycleGAN has two generators and two discriminators that are placed in a cyclic architecture (Zhu et al., 2017), unlike conditional generative adversarial networks (cGAN) that has one generator and one discriminator (see Figs. 2a and 4a). In de Haan et al. (2021), a transfer-learning style method was proposed that use load generators’ with the weights from generators that are trained on transforming from autofluerescence images to MT as per presented earlier in Zhang et al. (2020); de Haan et al. (2021). cGAN requires having pairs of real/virtual samples. Meanwhile in CycleGAN, each discriminator is trained to differentiate between a real sample with the virtual version of the same samples generated after passing through the two generators back and forth. CycleGAN, see Fig. 6, is a state-of-the-art method in terms of image transformation, because it has a superior capability in different applications where we cannot have paired training images, such as colorization and augmentation (Zhu, Park, Isola, Efros, 2017, Bashkirova, Usman, Saenko). We hypothesize that, if we have a paired HE-MT dataset, cGAN may be able to achieve more precise pixel-level estimation of MT appearance. To validate our hypothesis, we collected a set histopathological slides, and they were scanned to digital slides at magnification ratio. Based on the results presented in Levy et al. (2020), CycleGAN was successful in obtaining good virtualization of MT stain on magnification ratio, in spite of the unpaired training set. In this paper, we investigate how we can adapt the training data to be used in cGAN. We also compare the performance of using cGAN using the presented methods over CycleGAN. We support our study by having higher resolution at magnification ratio. We also conduct a group of experiments to illustrate the ability of the proposed system to generate realistic MT stain and to perform fibrosis detection and quantification in the produced MT WSIs, compared to CycleGAN learning style.
To partially overcome the limitations of the previous HE-to-MT transformation attempts, we propose a novel digital pathology system (see Fig. 2a for the training framework, and Fig. 2b for the real-time usage framework) that, first, produces virtual MT from HE slides and, second, automatically detects the footprint of fibrous tissues in the produced virtual MT. The proposed system features a comprehensive training pipeline (Fig. 2a) that includes a process for tissue MT restaining, and an automatic rigid-body feature-based WSI automatic registration algorithm. The registration algorithm is incorporated in our pipeline to ensure near-perfect alignment of training pairs. The transformation model cGAN can learn accurate pixel-level model from paired HE and MT training images at the same tissue section. The proposed system can improve the accuracy of fibrosis staging by enabling examination of each tissue sections under MT and HE simultaneously, unlike using different sections in the current protocol. The proposed system improves the efficiency of fibrosis staging by eliminating the time and cost required to prepare physical MT slides. Details of our pipeline are fully described in the next section.
Section snippets
Methods
The proposed In-Silico fibrosis detection and quantification system has four main components: (1) WSI Rigid-Body Registration component that produces the pixel-level paired HE-MT images required for Pix2Pix training, (2) Color Normalization component that augments the digital histology images and to normalize the color appearance variability between slides, (3) Domain Transformation component that uses the input HE images to produce the virtual MT images, (4) Fibrosis Detection and
Material and data collection
The material used in the experiments consist of liver tissue specimens collected from 5 human subjects during liver transplantation surgeries. The study obtained the required IRB approvals and follows research ethics protocols. We also collected two additional sets of 16 and 5 pairs of slides, from 16 and 5 human subjects to be used as validation sets. The validation sets is collected from independent subjects and were converted to digital slides in independent setups and using different WSI
Experimental results
We start with presenting the performance of the registration algorithm, and qualitative and quantitative performance of the proposed stain model. We then move to the results of the ablation study that address the different dimensions of the proposed system by altering normalization technique, patch size, and magnification ratio are given. This is followed by qualitative and quantitative comparison between the proposed system and CycleGAN (Levy et al., 2020). Finally, we present the capability
Discussion
In this paper, we investigate the applicability to use cGAN model to perform transformation from HE to virtual MT. Next we studied whether the produced virtual MT can be used to segment fibrous tissue segments as a step towards fibrosis staging in chronic liver disease. We performed a group of experiments in order to explore the dimensions of this problem and to evaluate the performance of the proposed system qualitatively and quantitatively.
The proposed system encapsulates a WSI rigid-body
Conclusion and future work
In this paper, we propose a novel digital pathology system that can detect and quantify fibrosis in HE WSI. The proposed system is trained using a comprehensive pipeline that includes cGAN based HE-to-MT transformation model, WSI registration algorithm, and color-based detection model. We compared the proposed system with the different approaches at which fibrosis can be segmented Using U-net based model either directly from HE, or from the produced virtual MT. According to the set of
Declaration of Competing Interest
Authors declare that they have no conflict of interest.
Acknowledgments
The research project was reviewed and approved by the institutional research board at University of Louisville under IRB no. 21.0753. Also, this research work was supported in part by the U.S. National Science Foundation (NSF) under grant CNS1828521. Some of the systems and methods presented in this research work were filed as part of U.S. Patent Application 63/213,628.
References (31)
- et al.
Speeded-up robust features (surf)
Comput. Vis. Image Underst.
(2008) - et al.
Reply to: correspondence on a new definition for metabolic associated fatty liver disease: an international expert consensus statement”: mafld: moving from a concept to practice
J. Hepatol.
(2020) - et al.
A new definition for metabolic associated fatty liver disease: an international expert consensus statement
J. Hepatol.
(2020) - et al.
Letter regarding a new definition for metabolic dysfunction-associated fatty liver disease: an international expert consensus statement”
J. Hepatol.
(2020) - et al.
The future of pathology is digital
Pathology-Research Pract.
(2020) - et al.
A new metric based on extended spatial frequency and its application to DWT based fusion algorithms
Inf. Fusion
(2007) - et al.
Performance analysis of cost and accuracy for whale swarm and RMSprop optimizer
IOP Conference Series: Materials Science and Engineering
(2020) - Bashkirova, D., Usman, B., Saenko, K., 2018. Unsupervised video-to-video translation. arXiv preprint...
- et al.
Towards virtual h&e staining of hyperspectral lung histology images using conditional generative adversarial networks
Proceedings of the IEEE International Conference on Computer Vision Workshops
(2017) - et al.
Anhir: automatic non-rigid histological image registration challenge
IEEE Trans. Med. Imaging
(2020)
Stochastic gradient descent tricks
Neural Networks: Tricks of the Trade
The diagnosis and management of nonalcoholic fatty liver disease: practice guidance from the american association for the study of liver diseases
Hepatology
Adaptive subgradient methods for online learning and stochastic optimization
J. Mach. Learn. Res.
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
Commun. ACM
Deep learning-based transformation of H&E stained tissues into special stains
Nat. Commun.
Cited by (5)
The regulatory role of cancer stem cell marker gene CXCR4 in the growth and metastasis of gastric cancer
2023, npj Precision Oncology