Abstract
Obtaining frozen sections of bone tissue for intraoperative examination is challenging. To identify the bony edge of resection, orthopaedic oncologists therefore rely on pre-operative X-ray computed tomography or magnetic resonance imaging. However, these techniques do not allow for accurate diagnosis or for intraoperative confirmation of the tumour margins, and in bony sarcomas, they can lead to bone margins up to 10-fold wider (1,000-fold volumetrically) than necessary. Here, we show that real-time three-dimensional contour-scanning of tissue via ultraviolet photoacoustic microscopy in reflection mode can be used to intraoperatively evaluate undecalcified and decalcified thick bone specimens, without the need for tissue sectioning. We validate the technique with gold-standard haematoxylin-and-eosin histology images acquired via a traditional optical microscope, and also show that an unsupervised generative adversarial network can virtually stain the ultraviolet-photoacoustic-microscopy images, allowing pathologists to readily identify cancerous features. Label-free and slide-free histology via ultraviolet photoacoustic microscopy may allow for rapid diagnoses of bone-tissue pathologies and aid the intraoperative determination of tumour margins.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$99.00 per year
only $8.25 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The main data supporting the findings of this study are available within the paper and its Supplementary Information. The training dataset and the fake output images for the CycleGAN network are available at https://doi.org/10.5281/zenodo.6345772. The raw data generated during the study are too large to be publicly shared, yet they are available for research purposes from the corresponding authors on reasonable request.
Code availability
The original code for CycleGAN is available at https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix. We applied this code to our dataset with the customized settings described in Methods. MATLAB was used for creating image tiles for the network and for restitching the output image tiles. The quantitative analysis of photoacoustic virtual histology was done via QuPath (https://qupath.github.io). The system control software and the data collection software are proprietary and used in licensed technologies.
References
Global Cancer Observatory (WHO, accessed 19 May 2021); http://gco.iarc.fr/today/home
Wyld, L., Audisio, R. A. & Poston, G. J. The evolution of cancer surgery and future perspectives. Nat. Rev. Clin. Oncol. 12, 115–124 (2015).
Sullivan, R. et al. Global cancer surgery: delivering safe, affordable, and timely cancer surgery. Lancet Oncol. 16, 1193–1224 (2015).
Mahe, E. et al. Intraoperative pathology consultation: error, cause and impact. Can. J. Surg. 56, E13–E18 (2013).
DiNardo, L. J., Lin, J., Karageorge, L. S. & Powers, C. N. Accuracy, utility, and cost of frozen section margins in head and neck cancer surgery. Laryngoscope 110, 1773–1776 (2000).
Brender, E., Burke, A. & Glass, R. M. Frozen section biopsy. JAMA 294, 3200 (2005).
Campanacci, M. Bone and Soft Tissue Tumors: Clinical Features, Imaging, Pathology and Treatment (Springer, 2013).
Pathology and Genetics of Tumours of Soft Tissue and Bone (IARC, WHO, 2002).
Taqi, S. A., Sami, S. A., Sami, L. B. & Zaki, S. A. A review of artifacts in histopathology. J. Oral Maxillofac. Pathol. 22, 279 (2018).
Gomez-Brouchet, A. et al. Assessment of resection margins in bone sarcoma treated by neoadjuvant chemotherapy: literature review and guidelines of the bone group (GROUPOS) of the French sarcoma group and bone tumor study group (GSF-GETO/RESOS). Orthop. Traumatol. Surg. Res. 105, 773–780 (2019).
Gareau, D. S. et al. Confocal mosaicing microscopy in Mohs skin excisions: feasibility of rapid surgical pathology. J. Biomed. Opt. 13, 054001 (2008).
Wang, M. et al. Gigapixel surface imaging of radical prostatectomy specimens for comprehensive detection of cancer-positive surgical margins using structured illumination microscopy. Sci. Rep. 6, 27419 (2016).
Wang, M. et al. High-resolution rapid diagnostic imaging of whole prostate biopsies using video-rate fluorescence structured illumination microscopy. Cancer Res. 75, 4032–4041 (2015).
Glaser, A. K. et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nat. Biomed. Eng. 1, 0084 (2017).
Fereidouni, F. et al. Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nat. Biomed. Eng. 1, 957–966 (2017).
Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat. Med. 26, 52–58 (2020).
Orringer, D. A. et al. Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy. Nat. Biomed. Eng. 1, 0027 (2017).
Assayag, O. et al. Large field, high resolution full-field optical coherence tomography: a pre-clinical study of human breast tissue and cancer assessment. Technol. Cancer Res. Treat. 13, 455–468 (2014).
Nguyen, F. T. et al. Intraoperative evaluation of breast tumor margins with optical coherence tomography. Cancer Res. 69, 8790–8796 (2009).
Fereidouni, F., Tracy, J. & Levenson, R. M. M. D. MUSE microscopy for thick tissue imaging with extended depth of field. In Proc. SPIE 10489, Optical Biopsy XVI: Toward Real-Time Spectroscopic Imaging and Diagnosis 104890H (SPIE, 2018).
Gambichler, T. et al. Comparison of histometric data obtained by optical coherence tomography and routine histology. J. Biomed. Opt. 10, 044008 (2005).
Wang, L. V. & Hu, S. Photoacoustic tomography: in vivo imaging from organelles to organs. Science 335, 1458–1462 (2012).
Wang, L. V. & Yao, J. A practical guide to photoacoustic tomography in the life sciences. Nat. Methods 13, 627–638 (2016).
Danielli, A. et al. Label-free photoacoustic nanoscopy. J. Biomed. Opt 19, 086006 (2014).
Shi, J., Tang, Y. & Yao, J. Advances in super-resolution photoacoustic imaging. Quant. Imaging Med. Surg. 8, 724–732 (2018).
Yao, J., Wang, L., Li, C., Zhang, C. & Wang, L. V. Photoimprint photoacoustic microscopy for three-dimensional label-free subdiffraction imaging. Phys. Rev. Lett. 112, 014302 (2014).
Yao, J. et al. High-speed label-free functional photoacoustic microscopy of mouse brain in action. Nat. Methods 12, 407–410 (2015).
Li, L. et al. Single-impulse panoramic photoacoustic computed tomography of small-animal whole-body dynamics at high spatiotemporal resolution. Nat. Biomed. Eng. 1, 0071 (2017).
Wong, T. T. W. et al. Fast label-free multilayered histology-like imaging of human breast cancer by photoacoustic microscopy. Sci. Adv. 3, e1602168 (2017).
Zhang, C., Zhang, Y. S., Yao, D.-K., Xia, Y. & Wang, L. V. Label-free photoacoustic microscopy of cytochromes. J. Biomed. Opt. 18, 020504 (2013).
Xu, Z., Li, C. & Wang, L. V. Photoacoustic tomography of water in phantoms and tissue. J. Biomed. Opt. 15, 036019 (2010).
Wong, T. T. W. et al. Label-free automated three-dimensional imaging of whole organs by microtomy-assisted photoacoustic microscopy. Nat. Commun. 8, 1386 (2017).
Shi, J. et al. High-resolution, high-contrast mid-infrared imaging of fresh biological samples with ultraviolet-localized photoacoustic microscopy. Nat. Photonics 13, 609–615 (2019).
Tschuchnig, M. E., Oostingh, G. J. & Gadermayr, M. Generative adversarial networks in digital pathology: a survey on trends and future potential. Patterns 1, 100089 (2020).
Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).
Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision 2223–2232 (2017).
Lahiani, A. et al. Virtualization of tissue staining in digital pathology using an unsupervised deep learning approach. In European Congress on Digital Pathology 47–55 (Springer, 2019).
Yao, D.-K., Chen, R., Maslov, K., Zhou, Q. & Wang, L. V. Optimal ultraviolet wavelength for in vivo photoacoustic imaging of cell nuclei. J. Biomed. Opt. 17, 056004 (2012).
Li, X., Kang, L., Zhang, Y. & Wong, T. T. W. High-speed label-free ultraviolet photoacoustic microscopy for histology-like imaging of unprocessed biological tissues. Opt. Lett. 45, 5401–5404 (2020).
Imai, T. et al. High-throughput ultraviolet photoacoustic microscopy with multifocal excitation. J. Biomed. Opt. 23, 036007 (2018).
Li, B., Qin, H., Yang, S. & Xing, D. In vivo fast variable focus photoacoustic microscopy using an electrically tunable lens. Opt. Express 22, 20130–20137 (2014).
Tang, M., Luo, F. & Liu, D. Automatic time gain compensation in ultrasound imaging system. In 2009 3rd International Conference on Bioinformatics and Biomedical Engineering 1–4 (IEEE, 2009); https://doi.org/10.1109/ICBBE.2009.5162432
Xu, Z. et al. Cortex-wide multiparametric photoacoustic microscopy based on real-time contour scanning. Neurophotonics 6, 035012 (2019).
Mao, X. et al. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision 2794–2802 (2017).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016).
Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 1125–1134 (2017).
Zhang, R. Making convolutional networks shift-invariant again. In International conference on machine learning 7324–7334 (2019).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://doi.org/10.48550/arXiv.1412.6980 (2017).
Humphries, M. P., Maxwell, P. & Salto-Tellez, M. QuPath: the global impact of an open source digital pathology system. Comput. Struct. Biotechnol. J. 19, 852–859 (2021).
Acknowledgements
We thank M. D’Apuzzo for helpful discussions and valuable pathological feedback. This work was sponsored by the United States National Institutes of Health (NIH) grants R01 CA186567 (NIH Director’s Transformative Research Award), R35 CA220436 (Outstanding Investigator Award) and R01 EB028277A.
Author information
Authors and Affiliations
Contributions
R.C., B.C. and L.V.W. designed the experiment. R.C. and Y.Z. built the system and wrote the control programme. R.C. performed the experiment. S.D.N., B.C. and Y. Liang provided bone specimens and H&E slices. R.C., S.D. and Y. Luo performed image processing. L.V.W. and B.C. supervised the project. All authors were involved in discussions during the work and in the preparation of the manuscript.
Corresponding authors
Ethics declarations
Competing interests
L.V.W. has a financial interest in MicroPhotoAcoustics, CalPACT and Union Photoacoustic Technologies. (However, these companies did not financially support this work.) The other authors declare no competing interests.
Peer review
Peer review information
Nature Biomedical Engineering thanks Ji-Xin Cheng, Carolin Mogler and Ashok Veeraraghavan for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary figures and tables.
Supplementary video 1
Side-by-side comparisons of greyscale UV-PAM and H&E images.
Supplementary video 2
Side-by-side comparisons of the virtually stained UV-PAM images and the corresponding H&E images in Fig. 7a,b.
Supplementary video 3
Side-by-side comparisons of the virtually stained UV-PAM images and the corresponding H&E images in Fig. 7c,d.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cao, R., Nelson, S.D., Davis, S. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. Biomed. Eng 7, 124–134 (2023). https://doi.org/10.1038/s41551-022-00940-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41551-022-00940-z
This article is cited by
-
Correction of high-rate motion for photoacoustic microscopy by orthogonal cross-correlation
Scientific Reports (2024)
-
Virtual histological staining of unlabeled autopsy tissue
Nature Communications (2024)
-
Fast Real-Time Brain Tumor Detection Based on Stimulated Raman Histology and Self-Supervised Deep Learning Model
Journal of Imaging Informatics in Medicine (2024)
-
Digital staining in optical microscopy using deep learning - a review
PhotoniX (2023)
-
Deep learning-enabled virtual histological staining of biological samples
Light: Science & Applications (2023)