Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism

Abstract

Deep learning-based methods have recently shown promising results in image inpainting. These methods generate patches with visually plausible image structures and textures, which are semantically coherent with the context of surrounding regions. However, existing methods tend to generate artifacts which are inconsistent with surrounding regions, especially when dealing with complex images. Aiming at the limitations current in deep learning-based methods, this paper proposes a multi-scale generative adversarial network model based on cross-layer attention transfer mechanism. Cross-Layer Attention Transfer Module (CL-ATM) is presented to guide the filling of the corresponding low-level semantic feature map by using the high-level semantic feature map, so as to ensure visual and semantic consistency of inpainting. Meanwhile, a multi-scale generator and the multi-scale discriminators are added into the network structure. Different scales of discriminators have different receptive fields, which enable the generator to produce images with better global consistency and more details. Qualitative and quantitative experiments show that our method has superior performance against state-of-art inpainting models.

Introduction

With the rapid development of deep learning in the field of computer vision, researches on image editing and image generation have achieved remarkable results. Image inpainting is a hot issue involving image editing and generation [1], [2], [3], which can also be extended to tasks including image un-cropping, rotation, stitching, re-targeting, re-composition, compression, super-resolution, harmonization and many computer vision problems [4], [5], [6], [7], [8], [9].

Image inpainting refers to fill the missing pixels in the damaged image [10] given corresponding masks, as shown in Fig. 1. Human can easily complete the image restoration based on the content of surrounding environment, but it is especially difficult for computer since there is no unique result and no universally applicable method. Currently, the key issue of inpainting is how to assist the model to make full use of existing information and how to judge whether results have enough authenticity and rationality.

Existing image inpainting works can be roughly divided into two categories: traditional methods [1], [11], [12] and deep learning-based methods [3], [13], [14], [15], as shown in Table 1. Traditional methods either match and replicate the background patches from low resolution to high resolution, or synthesize textures by propagating the neighbor region appearance to the target holes. These methods can synthesize detailed results and have been widely used in practice. Owing to the lack of advanced understanding of image semantics, traditional methods cannot produce semantically reasonable results. In recent years, the emergence of deep neural networks such as Convolutional Neural Network (CNN) [16], [17] and Generative Adversarial Network (GAN) [18] have greatly improved the performance of tasks such as classification, object detection, semantic segmentation and feature selection [19], [20], [21], [22], [23], [24]. Deep learning-based methods first encode the semantic context of images into potential feature spaces through deep neural network, and then generate semantically relevant patches through generating model. Meanwhile, the consistency between generated pixels and existing pixels is encouraged by the training with the adversarial network. However, the methods based on deep neural networks such as the context encoder (CE) [3] and global-local model (GL) [13] tend to produce boundary artifacts and distorted structures, which are mainly caused by the low efficiency of CNN in explicitly borrowing or reproducing the information from distant spatial locations. Some recent studies attempt to solve this problem by using the information of known regions to estimate the missing pixels. For example, the context attention layer [25] and patch-swap layer [26] proposed by Song et al. and Yu et al. respectively are used to fill in missing pixels with similar patches from undamaged regions at high-level feature maps. Nevertheless, the visual consistency of the repaired image remains a major challenge.

In order to ensure the consistency of visual effect and semantics, and to get high-quality results, we propose a novel multi-scale GAN with cross-layer attention transfer mechanism. Our model consists of two stages. The first stage network with dilated convolution simply reconstructs the missing areas by reconstruction loss. Cross-layer attention transfer mechanism is added in the second stage network. To begin with, a cross-layer attention transfer module (CL-ATM) is proposed. We calculate the cosine similarity between the patches of missing areas and the known areas in a high-level semantic feature map, and then obtain the attention score map through softmax. The attention score map is used to guide the transfer of the related features from known areas to missing areas in the lower-level feature map with higher resolution. Further, the second stage generator network uses deep feature maps to generate repaired images of different sizes, meanwhile, multi-scale GAN discriminators are added on the basis of the global discriminator and local discriminator. Multi-scale discriminators are trained on images with different resolutions, which can guide the generator to produce images with more details and better visual effects. The entire network is trained with two stages of reconstruction losses and four WGAN-GP losses.

In short, the main contributions of our work include:

1. We propose a multi-scale GAN consisting of one generator and four adversarial discriminators, which deploys the reconstruction and adversarial losses to synthesize the missing content from damaged images. The WGAN-GP loss and dilated convolution are used to improve the training stability and speed.

2. We introduce CL-ATM to utilize the attention score learned from the high-level semantic feature map. It can guide the feature transfer of the low-level semantic feature map in a corresponding layer (filling missing regions with known regions). It ensures the visual and semantic consistency of the generated image.

3. The experiments on Place2, CelebA, ImageNet, DTD, COCO, and Paris StreetView datasets have validated the effectiveness of our method. Compared with existing state-of-the-art methods, our method can generate higher quality images.