Neural texture transfer assisted video coding with adaptive up-sampling

Highlights

• We introduce reference-based SR in down/up-sampling-based video coding method, where target and reference images are not required to be texture-aligned as required in existing methods.

• We proposed an adaptive group of pictures (GOP) method to automatically decide the adaptive sampling scheme.

• The neural texture transfer model for reference-based SR produces realistic up-sampled frame at the decoding end.

Abstract

Deep learning techniques have been extensively investigated for the purpose of further increasing the efficiency of traditional video compression. Some deep learning techniques for down/up-sampling-based video coding were found to be especially effective when the bandwidth or storage is limited. Existing works mainly differ in the super-resolution models used. Some works simply use a single image super-resolution model, ignoring the rich information in the correlation between video frames, while others explore the correlation between frames by simply concatenating the features across adjacent frames. This, however, may fail when the textures are not well aligned. In this paper, we propose to utilize neural texture transfer which exploits the semantic correlation between frames and is able to explore the correlated information even when the textures are not aligned. Meanwhile, an adaptive group of pictures (GOP) method is proposed to automatically decide whether a frame should be down-sampled or not. Experimental results show that the proposed method outperforms the standard HEVC and state-of-the-art methods under different compression configurations. When compared to standard HEVC, the BD-rate (PSNR) and BD-rate (SSIM) of the proposed method are up to -19.1% and -26.5%, respectively.

Introduction

According to recent statistics, video traffic will account for about 82% of all Internet traffic by 2022 [1]. Video has become one of the major ways for information transmission and communication. At the same time, new video types, including Ultra-High Definition (UHD), Virtual Reality (VR), Wide Color Gamut (WCG), and High Dynamic Range (HDR), are emerging. These new video types provide a better user experience but at the cost of dramatically increased data volumes. Meanwhile, the number of video cameras in use keeps boosting in recent years, such as surveillance cameras, laptops, and smartphones. Consequently, the total amount of global video data doubles every two years, which has been the bottleneck for data processing, storage, and transmission [2]. Therefore, more advanced video compression techniques are of vital importance, which will support more efficient storage and transmission of videos.

During the last three decades, the development of traditional statistical video compression methods [3], [4], [5], [6], [7] has somewhat saturated and most recent endeavors turned to deep learning models [8], [9], [10], which have proved their capacity to discover knowledge from unstructured massive data and provide data-driven predictions. Deep learning has the potential to provide new opportunities for further upgrading video coding technologies. Inspired by the recent advances in deep learning, many works have been proposed to leverage deep learning in video compression and achieve significant improvements [11], [12]. Among these, Convolution Neural Network (CNN) based video enhancement was proposed as a post-processing procedure at the decoder to improve the perceptual quality of the reconstructed video [13], [14], [15]. To further increase the compression ratio, some works propose to down-sample the video prior to encoding and up-sample the decoded video using CNN-based video super-resolution model, which is known as down/up-sampling-based coding [16], [17], [18]. As reported in [19], [20], the coding of low-resolution video can perform both subjectively and objectively better than the direct coding of full-resolution version at low bit rates.

Prior work in down/up-sampling-based coding is mainly inspired by CNN-based single image super-resolution (SR), which outperforms traditional methods with a huge margin [21]. For example, a CNN-based up-sampling scheme is proposed in [16] for intra-frame coding. A video coding scheme is proposed in [22], where a group of pictures (GOP) is entirely down-sampled and compressed, and each frame is individually up-sampled using trained CNN models. Although these methods have demonstrated the potential of down/up-sampling-based coding with CNN for improving compression performance, they have not exploited the correlation between neighboring frames.

To this end, several attempts have been made to use the correlation between neighboring frames to further enhance the performance [17], [23], [24]. Lin et al. [25] proposed to adaptively divide frames into keyframes (KFs) and nonkey frames (NKFs), which are encoded at the original resolution and at a reduced resolution, respectively. At the decoder, NKFs are reconstructed with the corresponding motion estimation block in KFs using CNN. In addition to frame-level down/up-sampling-based coding, block-level down/up-sampling-based coding are also proposed to improve the performance [17], [26]. In [17], each block in the P/B frame can either be compressed at the original resolution or down-sampled and compressed at a lower resolution. At the decoder, low-resolution blocks are up-sampled by the CNN models. The block-based scheme provides the flexibility to deal with the spatially variant texture and motion characteristics in natural videos. However, it is quite computational intensive for block-level processing.

In this paper, frame-level down/up-sampling-based coding is employed to reduce the computational complexity. Meanwhile, a neural texture transfer-based frame up-sampling is proposed to cope with the spatially variant texture and motion characteristics in natural videos. As in the existing frame-level schemes [23], [25], features of the low-resolution frame and the full-resolution reference frame are simply concatenated, where only co-located features can be fused. However, the best matching feature is not necessarily co-located, thus leading to a sub-optimal result. While with neural texture transfer [27], a multi-level matching is conducted in the neural space, instead of the raw pixel space, to adaptively transfer texture from the reference images to the target image. This matching scheme facilitates the semantic texture transfer, which provides robust results even when irrelevant reference images are provided. To achieve optimized performance, the neural texture transfer model was fine-tuned on HEVC compressed video sequences. The proposed approach has been compared to both the HEVC anchor [28] and the block-level scheme [17], with results demonstrating consistent improvements on the HEVC common test sequences [29] for different QP ranges. Specifically, the main contributions of our work are as follows:

• We propose to use the semantic texture transfer for down/up-sampling-based video coding, which exploits the semantic correlation between the reference frames and the target frame, leading to significant enhancement of the frame-level SR performance.

• We propose a non-uniform compression scheme, where frames are adaptively compressed at the original or reduced resolution. Thus, frames encoded at the original resolution can be used as reference frames for the restoration of other frames.

• Our model outperforms the state-of-the-art block-level down/up-sampling-based coding scheme while requiring a lower computational complexity.

The rest of the paper is organized as follows. In Section 2, the deep learning-based up-sampling methods and down/up-sampling-based video coding methods are reviewed. Section 3 introduces the proposed neural texture transfer assisted frame-level down/up-sampling-based coding scheme, the CNN architecture of the neural texture transfer, and the training strategy. Section 4 describes the experiments and results. Finally in Section 6, conclusion and future work are presented.