Single image super-resolution via hybrid resolution NSST prediction

Highlights

• This is the first attempt to solve image SR as a NSST coefficients prediction problem.

• A new deep hybrid resolution network is proposed by a residual-in-residual style.

• The proposed NSST-based framework can be easily adapted to any arbitrary CNN architectures.

• Our method achieves more appealing results than previous SISR methods.

• We show that our proposed solutions are effective for real-world image SR.

Abstract

Convolutional neural networks (CNNs) have achieved great success in single image super-resolution (SR). However, most previous methods predict high-resolution (HR) images in the spatial domain, producing over-smoothed outputs while losing texture details. To address this problem, in this paper we propose to predict nonsubsampled shearlet transform (NSST) coefficients, which better represent the global topology information and local texture details of HR images. On the other hand, we propose a deep hybrid resolution network by a residual-in-residual style, which aggregates features of multiple resolutions so as to gather rich context information in compact representations. When evaluated on a newly released RealSR dataset and traditional simulated datasets, our method, namely hybrid resolution NSST prediction (HRNP), achieves more appealing results, w.r.t. PSNR and SSIM, than the state-of-the-art methods. Moreover, we find our HRNP is more capable of preserving complex edges and curves than other methods.

Introduction

In recent years, the restoration of high-resolution (HR) images from given low-resolution (LR) images has been widely investigated by many researchers in the community of multimedia. This task is defined as super-resolution (SR). Since image SR restores the high-frequency information, it is widely used in many applications, ranging from medical and satellite imaging to visual surveillance, where high-frequency details are greatly desired. In this paper, we tackle the specific problem of single image super-resolution (SISR).

Recent advances in SISR take advantage of the powerful representation ability of convolutional neural networks (CNNs). However, synthesizing LR images as the training data for CNNs is quite challenging as it is difficult to obtain LR–HR image pairs in real-world applications. Therefore, most of existing methods train and evaluate their models on simulated datasets, where the LR images are artificially generated by applying bicubic downsampling to their HR counterparts. Unfortunately, existing models trained on such simulated datasets are hard to generalize to practical applications since the degradation functions in real-world are much more complex. To approximate the realistic degradation function more accurately, the RealSR dataset (Cai et al., 2019) is released, in which the LR images are generated by using digital single lens reflex (DSLR) cameras for reflecting a realistic situation. In this paper, we adopt the RealSR dataset for training and evaluation, allowing our model is capable of the real-world image SR in practical applications.

On the other hand, most concurrent SISR methods (Dong et al., 2016b, Tai et al., 2017b, Zhang et al., 2018, He et al., 2019) use a pixel-wise loss in image space, enforcing the pixel-wise outputs become more and more similar to the ground-truth HR images in the iterative CNN training procedure. However, such approaches usually produce blurry and over-smoothed outputs, loosing texture details. In contrast, some efforts are made to solve the SISR problem in the transform domain, which is able to retain the contextual and textural information of an image at different levels. Instead of learning the mapping from LR to HR images in the spatial domain, these methods formulate the SR problem as transform coefficients prediction. For instance, discrete wavelet transform (DWT) has been explored for SR in both conventional frameworks (Demirel and Anbarjafari, 2011b, Demirel and Anbarjafari, 2011a, Hui and Lam, 2012) and deep networks (Kumar et al., 2017, Guo et al., 2017, Liu et al., 2018, Zhong et al., 2018, Huang et al., 2019, Deng et al., 2019). DWT effectively deals with the “point singularity” problem of one-dimension signal. However, a common limitation of DWT is that it cannot well represent the curves and edges of two-dimensional images because of its isotropic property (Do and Vetterli, 2005). In order to overcome the disadvantages of DWT, we propose to use nonsubsampled shearlet transform (NSST), which consists of the non-subsampled Laplacian pyramid transform and several different shearing filters (Li et al., 2015). NSST provides optimal approximation for a piecewise smooth function, and is also fast to implement. In Fig. 1, we compare the high-frequency coefficients of NSST and DWT, where we can clearly see that NSST represents the curvature more accurately. In this paper, we propose to formulate the SISR problem as the prediction of NSST coefficients, which are able to preserve richer structural information than DWT and to avoid artifacts. The contributions are summarized as follows:

(1) We propose a deep hybrid resolution network (HyRNet) by a residual-in-residual style, which fully exploits multi-resolution features so as to gather rich context information in compact representations, allowing our model to generate more plausible SR results.

(2) A novel NSST and CNN based approach, namely Hybrid Resolution NSST Prediction (HRNP), is proposed for image SR. To the best of our knowledge, this is the first attempt to solve image SR as a NSST coefficients prediction problem. In principle, our NSST prediction can be easily adapted to any arbitrary CNN architectures.

(3) We show that our proposed solutions are effective for real-world image SR via model analysis. When evaluated on the simulated and RealSR datasets, our method achieves more appealing results both quantitatively and qualitatively than previous state-of-the-art methods.

Rest of the paper is organized as follows: Section 2 briefly reviews the related works on CNN-based SISR methods and transform domain based SISR methods, respectively. Section 3 describes the proposed method in detail. Model analysis and comparisons with the state-of-the-art methods are presented in Section 4. Finally, concluding remarks are made in Section 5.