Gated Fusion Network for Degraded Image Super Resolution

Abstract

Single image super resolution aims to enhance image quality with respect to spatial content, which is a fundamental task in computer vision. In this work, we address the task of single frame super resolution with the presence of image degradation, e.g., blur, haze, or rain streaks. Due to the limitations of frame capturing and formation processes, image degradation is inevitable, and the artifacts would be exacerbated by super resolution methods. To address this problem, we propose a dual-branch convolutional neural network to extract base features and recovered features separately. The base features contain local and global information of the input image. On the other hand, the recovered features focus on the degraded regions and are used to remove the degradation. Those features are then fused through a recursive gate module to obtain sharp features for super resolution. By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process by avoiding learning the mixed degradation all-in-one and thus enhance the final high-resolution prediction results. We evaluate the proposed method in three degradation scenarios. Experiments on these scenarios demonstrate that the proposed method performs more efficiently and favorably against the state-of-the-art approaches on benchmark datasets.

Appendix

1.1 Network Configuration

Table 6 Configuration of the proposed network. The values in the skip row are layer names, indicating whose outputs are added to the outputs of the corresponding layers

We present the detailed configuration of the proposed network in Table 6, with respect to the four modules in the network: the deblurring module, SR feature extraction module, recursive gate module, and reconstruction module.

1.2 List of the Evaluated Methods

All the the evaluated methods in Sect. 4 are listed in Table 7.

Table 7 List of the evaluated methods in Sect. 4

1.3 Additional Visual Results

In this section, we present more qualitative comparisons on the LR-RESIDE in Fig. 12, which includes the combinations of the SR algorithm (Lim et al. 2017) and dehazing algorithms (He et al. 2011; Berman et al. 2016; Ren et al. 2018).

Fig. 12

More visual comparisons on the LR-RESIDE dataset. The evaluated methods include EDSR (Lim et al. 2017), DCP (He et al. 2011), NLD (Berman et al. 2016), and DGFN (Ren et al. 2018). The proposed model generates sharper HR images with more details

1.4 Additional Ablation Study and Analysis

To further demonstrate the importance of the dual-branch architecture and gate module, more ablation study and visual results are presented in this section. We first compare the restoration module with the state-of-the-art restoration methods to evaluate the performance contribution brought by the image restoration module. Then, the qualitative results of the ablation study are presented to demonstrate how other modules help to improve the performance.

1.4.1 Performance of Restoration Module

We provide the quantitative results from the state-of-the-art restoration methods and the proposed restoration module in Table 8. The restoration methods include deblurring algorithms [DeepDeblur Nah et al. (2017), DeblurGAN Kupyn et al. (2018), and SRN Tao et al. (2018)], dehazing algorithms [DGFN Ren et al. (2018), GCANet Chen et al. (2019), and PFFNet Mei et al. (2018)), and deraining algorithms (IDGAN Zhang et al. (2019), RESN Li et al. (2018d), and DID-MDN Zhang and Patel (2018b)]. Since these restoration methods are trained on the high-resolution images (GOPRO, RESIDE, Rain1200 datasets), we re-train the restoration module on the same high-resolution datasets for fair comparisons. As shown in Table 8, in none of these three datasets does our restoration module acquire the best results, while the proposed GFN still performs favorably on all the three datasets as shown in Tables 1, 2, 3 of the manuscript. Therefore, the favorable performance of the proposed method comes from the architecture designs, such as the dual-branch architecture and the gate module.

1.5 Effect of Dual-Branch Architecture and Gate Module

To further demonstrate the benefits of the dual-branch architecture and gate module, we present an example in Fig. 13. Figure 13b, c show the outputs of the restoration module $G_{res}$ and Model-1 ($G_{res}$ + $G_{base}$ + $G_{recon}$) in Fig. 11a. Since the artifacts in the $G_{res}$ are propagated to the $G_{base}$ and $G_{recon}$, the Model-1 generates less satisfactory results as shown in Fig. 13c. Figure 13d shows the output of the Model-4 in Fig. 11d, which adopts the dual-branch architecture without the gate module $G_{gate}$. Figure 13d contains fewer artifacts than Fig. 13c, especially on the regions that are relatively sharper in the input image. This is because the dual-branch architecture combines features from both input images and recovered images and, therefore, avoids error propagation from only the recovered images. Fig. 13e shows the output of the proposed GFN introducing the gate module to adaptively fuse the features. By exploiting the confidence of the features from two branches ($\phi _{RF}$ into $\phi _{BF}$), the gate module manages to suppress the artifacts and blurry features via local and channel-wise feature fusion. Figure 13f–j shows that our model progressively fuses features and suppresses artifacts through the gate module.

Table 8 Quantitative comparison with the state-of-the-art restoration methods on three applications

Fig. 13

Qualitative results of the ablation study. $\phi _{BF}$ denotes the base features from the base module $G_{base}$ and $\phi _{RF}$ denotes the features from the restoration module $G_{res}$. All the models are trained on the LR-GOPRO dataset with the same training settings as the proposed GFN

1.6 Applications on Detection Task

To demonstrate that the proposed method can help the following high-level tasks, we compare the proposed GFN with state-of-the-art methods on the object detection task. We first generate two datasets from the KITTI dataset (Geiger et al. 2012), one blurry low-resolution dataset and a hazy low-resolution dataset. For the blurry dataset, we apply the single image non-uniform blurry synthesis method in Lai et al. (2016) to generate the blurry HR images and use the bicubic downsampling to generate the blurry LR images as the inputs. We then generate recovered HR images with the following methods: the bicubic upsampling, deblurring method SRN (Tao et al. 2018 with super resolution method RCAN (Zhang et al. 2018b), joint restoration and super-resolution method ED-DSRN (Zhang et al. 2018a), and the proposed GFN. For the hazy dataset, we first apply the single image depth estimation method, the Monodepth2 (Godard et al. 2019), to predict a depth map for each image and then synthesize the hazy image following the instruction of the RESIDE dataset (Li et al. 2018a). We compare the proposed GFN with the following approaches: the bicubic upsampling, dehazing method PFFNet (Mei et al. 2018) with super resolution method RCAN (Zhang et al. 2018b), and joint restoration and super-resolution method ED-DSRN (Zhang et al. 2018a). We use the above methods to recover HR images and then use the YOLOv3 (Redmon and Farhadi 2018) to evaluate the detection accuracy.

We show the detection accuracy in Tables 9 and 10. The HR images restored from the proposed GFN obtain the best detection accuracy in both applications. The qualitative results in Figs. 14 and 15 demonstrate that our GFN not only generates clean HR outputs but also improves the detection algorithm to recognize the cars and pedestrians.

Table 9 Objects detection results on the KITTI detection dataset (Geiger et al. 2012) with non-uniform motion blur

Table 10 Objects detection results on the KITTI detection dataset (Geiger et al. 2012) with haze degradation

Fig. 14

Detection results using the recovered images from different methods. We compare the following methods: bicubic upsampling, deblurring method SRN (Tao et al. 2018) + super resolution method RCAN (Zhang et al. 2018b), joint restoration and super-resolution method ED-DSRN (Zhang et al. 2018a), and the proposed GFN

Fig. 15

Detection results using the recovered images from different methods. We compare the following methods: bicubic upsampling, dehazing method PFFNet (Mei et al. 2018) + super resolution method RCAN (Zhang et al. 2018b), joint restoration and super-resolution method ED-DSRN (Zhang et al. 2018a), and the proposed GFN