A trusted medical image super-resolution method based on feedback adaptive weighted dense network

Highlights

• We introduce Feedback Adaptive Weight Dense Network (FAWDN) for medical image SR.

• A feedback mechanism is proposed to obtain trusted high-resolution images.

• We introduce adaptive weights into the dense links to adaptively select features.

Abstract

High-resolution (HR) medical images are preferred in clinical diagnoses and subsequent analysis. However, the acquisition of HR medical images is easily affected by hardware devices. As an effective and trusted alternative method, the super-resolution (SR) technology is introduced to improve the image resolution. Compared with traditional SR methods, the deep learning-based SR methods can obtain more clear and trusted HR images. In this paper, we propose a trusted deep convolutional neural network-based SR method named feedback adaptive weighted dense network (FAWDN) for HR medical image reconstruction. Specifically, the proposed FAWDN can transmit the information of the output image to the low-level features by a feedback connection. To explore advanced feature representation and reduce the feature redundancy in dense blocks, an adaptive weighted dense block (AWDB) is introduced to adaptively select the informative features. Experimental results demonstrate that our FAWDN outperforms the state-of-the-art image SR methods and can obtain more clear and trusted medical images than comparative methods.

Introduction

Medical images are widely used for clinical diagnosis, in which high-resolution (HR) images are preferred because they can provide much more significant structure and texture details than low-resolution (LR) images [1]. However, the acquisition of HR medical images is limited by hardware devices. As an effective and trusted alternative method, single image super-resolution (SR) aims to reconstruct the HR image from its LR counterpart. Image SR is initially proposed for natural image. Lots of medical image SR methods have also emerged in recent years, including interpolation-based [2], [3], reconstruction-based [4], [5], and learning-based methods [6], [7]. Although the interpolation-based SR methods are fast, the reconstructed HR image of these methods is blurred. The reconstruction-based methods consider image priors, e.g., non-local [8] and self-similarity [9], but these methods always lead to a lot of performance drop when the prior is inconsistent with test images. Conventional learning-based methods, e.g., sparse representation [6], compressed sensing [10], and random forest [11] have limited representation ability. Overall, the reconstructed HR images obtained by above methods are blurred or unreliable because these methods are susceptible to priors. Recently, convolutional neural network (CNN) based SR methods have achieved significant improvement in the performance. Dong et al. [12] propose the pioneering CNN-based method, which uses a full convolution neural network for natural image SR. After that, several SR networks focus on deeper or wider network architecture design such as VDSR [13], DRCN [14], SRDenseNet [15], and DRRN [16]. In addition to the CNN based methods, to obtain appealing visual effect, perceptual loss [17] and generative adversarial network (GAN) [18] are introduced into image SR. However, the textures and details generated by GAN may be different from the ground truth. In medical image SR, numbers of deep learning-based medical image SR methods have also been presented recently. Pham et al. [19] extend SRCNN to 3D cases for brain MR images SR. As for 2D medical image SR, Wei et al. [20] propose a deep network (DDSR) constructed by dense blocks for MR and CT image SR. However, all of the above methods are feedforward networks, in which the information can only pass from the input to the output. It is known that there always exists difference between the reconstruction result and the ground truth. In the feedforward network, its output cannot be transmitted to the input for improving the performance. As a consequence, these methods have limited ability to reconstruct fine textures, especially for the images with complicated details and textures.

To address the limitation of feedforward networks, we introduce a feedback mechanism into SR networks and propose a network denoted as Feedback Adaptive Weighted Dense Network (FAWDN) for trusted medical image SR. The implementation of this feedback mechanism is based on recurrent neural networks (RNNs) since RNN is indispensable for the feedback mechanism [21]. Through the feedback mechanism, our FAWDN can correct the error produced in the preceding time step and obtain a clearer HR image step by step.

As shown in Fig. 1, our FAWDN is comprised of three parts: the input unit, the hidden unit, and the output unit, respectively. The input unit is utilized to extract low-level features as the input state. Then, the concatenation of the input state and the hidden state will be fed into hidden unit to obtain a new hidden state, while the initial hidden state is the same as the input state. At each time step, the hidden state is passed to output unit to reconstruct HR images. Therefore, a reconstructed HR image is outputted by the FAWDN every time step. These reconstructed images have the same size but will be clearer with the time step. Besides, it will also be transmitted to the hidden unit at the next time step as the feedback signal to achieve the feedback mechanism. To obtain a superior hidden state, an adaptively weighted dense block (AWDB) is introduced into the hidden unit. AWDB is built on dense links [22], [15] because dense links have features reusing and strong representation capability. Furthermore, the dense link can eliminate gradient vanishing. However, in traditional dense links, each convolutional layer equally receives features from all previous convolutional layers, which makes the input features of each convolutional layer extraordinarily redundant. Therefore, we improve the original dense links by adding an adaptive weighting group before every convolutional layer in AWDB. So the convolutional layer can adaptively select informative features and reduce the feature redundancy. In summary, the contributions of this paper are as follows:

• To improve medical images resolution, we introduce FAWDN for medical image SR as an alternative method. Experimental results indicate that the introduced FAWDN could obtain superior SR performance over comparative image SR methods.

• To enhance the quality of reconstructed HR images, a feedback mechanism based on RNN is presented to correct the errors produced by the network at preceding time steps.

• To reduce the redundancy of input features of convolutional layers in dense link, we introduce adaptive weighting groups into AWDB to adaptively select informative features. The adaptive weighting groups can be trained with the FWADN.