Siamese CNN-based rank learning for quality assessment of inpainted images

Highlights

• Siamese CNN-based NR-IIQA, automatically ranking two inpainted images.

• A patch-wise coherence assessment module to evaluate local color and texture consistency within and around inpainted contents.

• A dataset containing thousands of inpainted image pairs with ground-truth ranking labels.

Abstract

Existing NR-IIQA (no reference-based inpainted image quality assessment) algorithms assess the quality of an inpainted image via artificially designed unnaturalness expression, which often fail to capture inpainted artifacts. This paper presents a new deep rank learning-based method for NR-IIQA. The model adopts a siamese deep architecture, which takes a pair of inpainted images as input and outputs their rank order. Each branch utilizes a CNN structure to capture the global structure coherence and a patch-wise coherence assessment module (PCAM) to depict the local color and texture consistency in an inpainted image. To train the deep model, we construct a new dataset, which contains thousands of pairs of inpainted images with ground-truth quality ranking labels. Rich ablation studies are conducted to verify the key modules of the proposed architecture. Comparative experimental results demonstrate that our method outperforms existing NR-IIQA metrics in evaluating both inpainted images and inpainting algorithms.

Introduction

Image inpainting is the task of filling holes in images caused by content damaging or unwanted object removing [1], [2], [3]. Image inpainting has received considerable attention and numerous effective inpainting algorithms have been proposed in the past decades. In this paper, we focus on inpainted image quality assessment (IIQA), which is essential in comparison of different inpainting methods.

There are already many IIQA methods. Some of them use fidelity metrics computed by referring to ground-truth images. These metrics, such as structural similarity index measure (SSIM) [4] and peak signal to noise ratio (PSNR), calculate the similarity between inpainted images and their ground-truth references, as the quality scores of the inpainted images. These metrics are precise but lack of practicality, because the original ground-truth images generally do not exist under non-experimental environments. Thereafter, many visual saliency-based IIQA metrics [5] were proposed to evaluate inpainted images. Although they require no ground-truth references, these methods tend to confuse visual salient regions with inpainted unnatural artifacts.

Recently, many no reference (NR) machine learning-based algorithms have been developed. For example, Viacheslav et al. [6] predicted an absolute quality score of an inpainted image via extracting artificially designed features to depict inpainted-and-surrounding inconsistency and training a support vector regression (SVR) model for feature-to-score mapping. Training these algorithms needs sufficient data with accurate ground-truth quality scores. However, it usually costs too much manual effort to score images. In addition, it is hard to provide objective quality scores: the scores given for the same image by different persons or even the same person at different moments are much different. Due to the above reasons, the learning-to-score NR-IIQA algorithms are hard to use in practice.

Considering that the main target of IIQA is to compare inpainted results, NR-IIQA can be solved by ranking, i.e. predicting the preference order of inpainted images. Compared to scoring inpainted images, the ground-truth data for training rank models can be easily obtained even without manual intervention [7], [8]. Unfortunately, existing learning-to-rank IIQA metrics are simply based on artificially designed features and traditional support vector machine (SVM) models [7], [8], which are much limited in inpainted-and-surrounding inconsistency expression and evaluation.

In this paper, we regard IIQA as a pairwise rank learning task as well and propose a new deep NR-IIQA algorithm for the assessment of inpainted images. The proposed model adopts a siamese CNN architecture trained for ranking two inpainted images. Each branch of CNN is designed and trained to encode the inpainted-and-surrounding inconsistency in global and multi-scale local perceptual levels. Training the proposed deep model requires sufficient data. Unfortunately, there is no large public IIQA datasets. In this paper, we construct a pairwise ranking dataset for the task of deep learning-to-rank IIQA. Within this dataset, we provide the quality ranking order of a pair of input inpainted images as ground-truth, which are much easier than manually annotating absolute scores to each training image. Once our model is well trained, it can judge which one in the input pair is better restored, and therefore can be used to rank the performance of inpainting algorithms, two by two.

Our main contributions are as follows:

• We propose a siamese CNN-based algorithm for NR-IIQA. To the best of our knowledge, this is the first deep learning-based IIQA model. The proposed model can automatically rank two inpainted images by capturing the multi-grain perceptual consistency within and around inpainted contents.

• To address the lack of training data for deep learning-to-rank IIQA, we synthesize a large dataset, containing thousands of inpainted image pairs. We also construct a relatively small real dataset with images repaired by various inpainting algorithms. Our model is trained with the synthetic data plus around half of the real data and tested in the left part of the real dataset.

• We conduct rich ablation studies to verify the key modules of the proposed model. We also compare our model with both reference-based metrics and no reference-based methods. Experimental results demonstrate that our model achieves performance closest to reference-based metrics and outperforms existing no reference-based metrics by large margins. In addition, we apply the proposed model to evaluate different inpainting algorithms, in which the proposed model obtains performance closest to the reference-based PSNR metric.

The rest of the paper is organized as follows. Section 2 discusses related works on IIQA metrics and algorithms. The constructed inpainted image dataset with pairwise quality ranking labels is introduced in Section 3. The details of the proposed deep learning-to-rank model for IIQA is described in Section 4. The performance and applications of the proposed model are presented in Section 5. Finally, Section 6 concludes the paper.