Model-data-driven image reconstruction with neural networks for ultrasound computed tomography breast imaging
Introduction
Ultrasound computed tomography (USCT) promises high specificity for early breast cancer detection without the associated risks of ionizing radiation while with lower costs than magnetic resonance tomography. It images the reflectivity, speed of sound (SoS) and attenuation of breast tissue. USCT devices allow to observe reflection and transmission tomography at the same time [1]. Yet, the widespread use of high-quality USCT is mainly limited by the excessive time for image reconstruction, which prevents currently the application of sophisticated reconstruction techniques within clinical workflows. Hence, there is an urgent need in fast and accurate reconstruction methods.
The simplified relationship of USCT between SoS, attenuation and breast tissue types is shown in Fig. 1. Both in combination of SoS and attenuation are expected to be a discriminator for different tissue types and also to distinguish malign and benign lesions [2]. The goal of USCT image reconstruction is to reconstruct SoS and attenuation, which are incorporated in a complex variable that describes the deviation of SoS and attenuation in the breast from the background medium of water [1].
The inverse problem of image reconstruction can be formulated as reconstruction of the inhomogeneity (ground truth) from ultrasound pressure field data (frequency data) measured from the USCT system, where
Here, X is the image domain; Y is the frequency domain; denotes the noise in the frequency data; the forward operator models how the ground truth gives rise to frequency data.
Recently, deep learning (DL) seems to offer new potentials in solving ill-posed inverse problems. Compared with traditional optimization-based algorithms, once learned, the inverse operators yield impressive reconstruction performance. Yet, the learning process is the Archilles’ heel of these techniques since the larger the data dimension is, the larger should be the necessary learning data for being rich enough to account for accurate mapping.
Many algorithms based on deep neural networks (DNNs) have been used for computed tomography (CT) image reconstruction. In particular, a convolutional encoder-decoder (CED) architecture [4] has been readily available for CT image reconstruction. The CED has been used to restore high-quality CT images from low dose CT images. Han et al. [5] propose a deep residual learning network for sparse view CT reconstruction via persistent homology analysis. Jin et al. [6] propose a filtered back projection conversion network (FBPConvNet) architecture which combines filtered back projection and U-net architecture to improve the image quality. In [7], a DD-Net based on DenseNet and deconvolution is developed to accelerate the network training speed.
However, using DL to reconstruct USCT images is rarely found in literature compared to CT reconstruction. From the mathematical side, the main difference is that CT can be well approximated by straight rays whereas USCT is a typical example of wave and scatter tomography, both complications that make standard CT reconstruction and other image reconstruction methods with DL not usable and pose severe problems in the reconstruction algorithms. It is a good starting point to mention the CT deep learning reconstruction as a motivation, but we cannot learn so much for USCT due to its wave properties (refraction, diffraction and scattering).
In this paper, we design a novel DNN architecture based on a primal–dual method [8] to obtain both fast and high-quality image reconstruction as a major breakthrough for these ultrasound applications. It maps the traditional primal-dual algorithm [8] into a dual-domain (pressure field frequency domain and image domain) network architecture for optimizing a general USCT image reconstruction model with a DNN.
In the traditional primal-dual hybrid gradient (PDHG) methods [8], [9], [10], a forward model and its adjoint are needed for the computation of each iteration. However, the forward model for USCT requires solving a wave equation [1], [11], which is a highly computational burden. In addition, computing the adjoint is time-consuming as well. Therefore, we replace the adjoint by our frequency-to-image domain network (FI-Net) architecture. Our network unrolls a fixed number of iterations of a primal-dual optimization strategy, where in each iteration a convolutional neural network (CNN) is applied for both frequency domain and image domain. Iterations are connected by both the forward operator and the FI-Net. In particular, in the image domain, nonlinear transforms are learned instead of learning traditional linear transforms. Furthermore, our network can be seen as an attempt to build a bridge between the model-based and data-based methods.
To gain substantial improvements and address above problems, we make the following contributions:
- •
The network is able to reconstruct images without any initial reconstruction such as filtered back-projection (FBP) and use CNNs to learn domain transformation without any fully connected layer. It can effectively reduce the number of parameters when compared with AUTOMAP [12], which makes it possible for large images, offering a powerful tool for solving reconstruction problems in clinically acceptable time.
- •
Due to the highly computational burden of adjoint, we propose the FI-Net to learn the adjoint from training data rather than computing it directly. Since the computation of FI-Net is much faster than the computation of the adjoint, the training of our proposed network would be hence much faster than the learned PDHG algorithm [13] which needs to compute the adjoint at every unrolled iteration in its network. For the same reason, our trained network would be much faster to use for image reconstruction than the learned PDHG network. Therefore, our network will benefit the community by the various and ubiquitous uses in different reconstruction problems, which need an adjoint as input.
Section snippets
Forward model
For conventional iterative image reconstruction, it is necessary to compute the forward model and back - propagation. In USCT, the basis for forward model is the wave propagation of ultrasound which is mathematically described by the wave equation in the frequency domain for inhomogeneous object [1].
Related work
We usually divide existing image reconstruction methods into two categories: model-based methods and data-based methods. In the following section, we will briefly review the two types of methods.
Methodology
In order to combine the advantageous properties of model-based and data-based methods and to overcome their limitations, we propose a hybrid model-data-driven image reconstruction approach. The goal is to design a deep neural network architecture that directly and quickly reconstructs high-quality images from USCT data without computation of adjoints. To the best of our knowledge, this is also the first work in the field of USCT image reconstruction based on DL.
Experimental design
We evaluate the algorithm on USCT problems with different methods. All the compared approaches are evaluated on the same Optical and Acoustic Breast Phantom Database (OA-Breast dataset), described below in more detail.
Discussion
The main focus of this paper is to introduce a model- and data- driven framework for USCT reconstruction and demonstrate its benefits when compared with traditional methods. Furthermore, our framework does not rely on any initial reconstruction as input: it can reconstruct images directly from frequency domain to image domain.
Compared with the state-of-the-art reconstruction methods, in particular for noisy data, our algorithm outperforms classical reconstruction algorithms by large SSIM and
Conclusion
In this paper, we present a model-data-driven image reconstruction method for USCT images. The algorithm is inspired by the PDHG algorithm, where we use FI-Net in the primal space in order to reduce the time for computing adjoint of derivative of forward operator. The algorithm is neither a typical model-driven method nor a purely data-driven method: it incorporates these two frameworks. Although it requires to apply different forward operators in different applications, and we generally do not
CRediT authorship contribution statement
Yuling Fan: Conceptualization, Methodology, Software, Validation, Writing - original draft, Visualization. Hongjian Wang: Conceptualization, Methodology, Formal analysis, Supervision, Writing - review & editing. Hartmut Gemmeke: Resources, Investigation, Data curation. Torsten Hopp: Resources, Investigation, Data curation. Juergen Hesser: Supervision, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
This work was supported by “the Fundamental Research Funds for the Central Universities” from Donghua University under grants No. 2232020D-36, the Donghua University Initial Research Funds for Young Teachers under grants No.112-07-0053079, the Deutsche Forschungsgemeinschaft (DFG) under grants No. HE 3011/37-1 and HO 5565/2-1. The authors acknowledge the authors of [13], Washington University in St. Louis for the source data and the models.
Yuling Fan received her M.E. degree in College of Information Engineering from Northwest A & F University in 2018. She is currently pursuing Ph.D. degree in the Faculty of Mathematics and Computer Sciences of Heidelberg University. Her major research interests include tomographic image reconstruction and deep learning.
References (45)
- et al.
3D ultrasound computer tomography for medical imaging
Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment
(2007) - et al.
Wave equation based transmission tomography
- et al.
Breast ultrasound tomography versus MRI for clinical display of anatomy and tumor rendering: preliminary results
American Journal of Roentgenology
(2012) - et al.
Clinical imaging with transmissive ultrasonic computerized tomography
IEEE Transactions on Biomedical Engineering
(1981) - et al.
LEARN: Learned experts assessment-based reconstruction network for sparse-data CT
IEEE Transactions on Medical Imaging
(2018) - Y.S. Han, J. Yoo, J.C. Ye, Deep residual learning for compressed sensing CT reconstruction via persistent homology...
- et al.
Deep convolutional neural network for inverse problems in imaging
IEEE Transactions on Image Processing
(2017) - et al.
A sparse-view ct reconstruction method based on combination of densenet and deconvolution
IEEE Transactions on Medical Imaging
(2018) - et al.
A first-order primal-dual algorithm for convex problems with applications to imaging
Journal of Mathematical Imaging and Vision
(2011) A primal–dual hybrid gradient method for nonlinear operators with applications to MRI
Inverse Problems
(2014)
Convex optimization problem prototyping for image reconstruction in computed tomography with the Chambolle-Pock algorithm
Physics in Medicine & Biology
Image reconstruction by domain-transform manifold learning
Nature
Learned primal-dual reconstruction
IEEE Transactions on Medical Imaging
Accelerating image reconstruction in ultrasound transmission tomography using L-BFGS algorithm
Solving ill-posed inverse problems using iterative deep neural networks
Inverse Problems
Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization
Physics in Medicine & Biology
A fast iterative shrinkage-thresholding algorithm for linear inverse problems
SIAM Journal on Imaging Sciences
An adaptive-ADMM algorithm with support and signal value detection for compressed sensing
IEEE Signal Processing Letters
ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing
Reconnet: Non-iterative reconstruction of images from compressively sensed measurements
Cited by (17)
Deep-Learning-Based Ultrasound Sound-Speed Tomography Reconstruction with Tikhonov Pseudo-Inverse Priori
2022, Ultrasound in Medicine and BiologyCitation Excerpt :In line with their work, Fan et al. (2021) proposed a dual domain network to pre-process the measured pressure data with a U-shape measurement domain network and then reconstruct the image using an image domain network with several dense blocks. Furthermore, Fan et al. (2022) combined the previous model-based and deep learning–based methods and proposed a model data-driven USST reconstruction method. These previous deep learning–based reconstruction methods had achieved very good reconstruction quality, but most of them used frequency-dependent pressure data.
ENCODER NEURAL NETWORK IN 2D ACOUSTIC TOMOGRAPHY
2024, Applied and Computational MathematicsLearned Full Waveform Inversion Incorporating Task Information for Ultrasound Computed Tomography
2024, IEEE Transactions on Computational ImagingSpatial and channel attention-based conditional Wasserstein GAN for direct and rapid image reconstruction in ultrasound computed tomography
2024, Biomedical Engineering Letters
Yuling Fan received her M.E. degree in College of Information Engineering from Northwest A & F University in 2018. She is currently pursuing Ph.D. degree in the Faculty of Mathematics and Computer Sciences of Heidelberg University. Her major research interests include tomographic image reconstruction and deep learning.
Hongjian Wang received the Ph.D. degree in computer science from the Systems and Transportation Laboratory of the Research Institute on Transportation, Energy and Society, University of Technology of Belfort-Montbéliard, Belfort, France, in 2016. From 2016 to 2019, he was a postdoctoral researcher with the Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany. Since 2020, he has been an associate professor with the School of Computer Science and Technology, Donghua University, Shanghai, China. His current research focuses on image reconstruction in ultrasound computed tomography.
Hartmut Gemmeke was born in Northeim, Germany, on March 9th, 1944. He studied exp. Physics in Göttingen and Heidelberg. Since 1990, he was a supernumerary professor at the University of Heidelberg and 1996 at the University of Karlsruhe. Since 2001 he was the director of the Institute for Data Processing and Electronics at KIT. He has about 400 publications including 25 in medical techniques. After his retirement 2010 he is working as scientific advisor at KIT. His present research activities are focused on the investigation of 3D Ultrasound Computed Tomography for early breast cancer recognition.
Torsten Hopp studied applied computer science at the University of Cooperative Education in Mannheim and the University of Heidelberg. He received his BSc. in 2006 and his MSc. in 2009. In 2012 he received his PhD in computer science from University of Mannheim. Since 2003 he is working at Karlsruhe Institute of Technology in the Institute for data processing and electronics. Since 2012 he is a postdoc and senior scientist in the Ultrasound Computer Tomography project. His main research interests cover medical imaging with ultrasound tomography, image reconstruction and medical image processing, including model-based image registration.
Juergen Werner Hesser is professor for Data Analysis and Modeling in Medicine at the Mannheim Institute for Intelligent Systems in Medicine (MIISM), Faculty of Medicine Mannheim, Heidelberg University since 2019. Before he was professor for Experimental Radiation Oncology at the same faculty since 2007. Between 2005 and 2007 he was Professor for Medical Technology, Institute of Computer Engineering, University of Mannheim. He habilitated 1999 in Computer Science, and earned his PhD and Diploma in Physics at Heidelberg University in 1992 resp. 1989. He is co-opted member of the Faculty of Physics and Astronomy and external member of the Central Institute for Computer Engineering (ZITI), and the Interdisciplinary Center for Scientific Computing (IWR) where he is member of the board of directors since 2020.