Learning to predict the quality of distorted-then-compressed images via a deep neural network

Highlights

• A stand-alone NR-IQA model is proposed for the distorted-then-compressed scenario.

• It is straightforward to accommodate hybrid distortions with easily-accessible data.

• A self-collected set bypasses the expensive and time-consuming subjective testing.

• The proposed model can be trained and make predictions in an opinion-unaware manner.

• The two-step methodology is realized in a unified framework without other metrics.

Abstract

Being captured by amateur photographers, reciprocally propagated through multimedia pipelines, and compressed with different levels, real-world images usually suffer from a wide variety of hybrid distortions. Faced with this scenario, full-reference (FR) image quality assessment (IQA) algorithms can not deliver promising predictions due to the inferior references. Meanwhile, existing no-reference (NR) IQA algorithms remain limited in their efficacy to deal with different distortion types. To address this obstacle, we explore a NR-IQA metric by predicting the perceptual quality of distorted-then-compressed images using a deep neural network (DNN). First, we propose a novel two-stream DNN to handle both authentic distortions and synthetic compressions and adopt effective strategies to pre-train the two branches of the network. Specifically, we transfer the knowledge learned from in-the-wild images to account for authentic distortions by utilizing a pre-trained deep convolutional neural network (CNN) to provide meaningful initializations. Meanwhile, we build a CNN for synthetic compressions and pre-train it on a dataset including synthetic compressed images. Subsequently, we bilinearly pool these two sets of features as the image representation. The overall network is fine-tuned on an elaborately-designed auxiliary dataset, which is annotated by a reliable objective quality metric. Furthermore, we integrate the output of the authentic-distortion-aware branch with that of the overall network following a two-step prediction manner to boost the prediction performance, which can be applied in the distorted-then-compressed scenario when the reference image is available. Extensive experimental results on several databases especially on the LIVE Wild Compressed Picture Quality Database show that the proposed method achieves state-of-the-art performance with good generalizability and moderate computational complexity.

Introduction

Overall mobile data traffic is expected to grow to 77 exabytes per month by 2022, a sevenfold increase over 2017 [1]. Online service providers like Netflix, YouTube, and Facebook are ingrained in the fabric of people’s life. Enormous visual signals are making their way of acquisition, transmission, and storage to the end-users through mobile devices, high definition television (HDTV), large photo-centric social networking websites, etc. With such a great load of images, quality degradation and perceptual information loss by ubiquitous distortions inevitably exist in these image communication and processing systems, which may exhibit a certain level of annoyance in the viewing experience. However, assessing the perceptual quality merely by humans is impractical for its time-consuming organization, troublesome criterion formulation, and expensive costs. Hence, objective image quality assessment, which measures perceived visual quality automatically by mimicking human perception, has become an effective substitution of subjective methods and is desirable in a wide range of computer vision applications.

According to the degree of dependence on the original information, objective IQA metrics can be generally classified into three mainstream categories: FR, reduced-reference (RR), and NR [2], [3], [4]. With access to high quality reference images like LIVE [5], CSIQ [6] and TID2013 [7] databases, prevalent FR models [8], [9], [10] perform reliably in measuring the perceptual quality of the distorted images. Nonetheless, FR approaches have limited use in most real-world scenarios [11] where neither full nor partial prior knowledge of high-quality references is available. In contrast, NR models do not rely on source information to assess the perceptual quality of images and hence attract a significant amount of research interests.

Particularly, there is a common scenario where hundreds of billions of user-generated images with imperfect quality (pre-distorted images) are continually uploaded onto social media and subsequently compressed (re-compressed images). Previously, we are all aware that the digitization, compression, storage, transmission, and display processes introduce modifications to the original image. But little attention has been paid to the acquisition stage which is affected by several factors such as lens limitations, aperture, lighting, and noise sensitivity. In addition, casual, inexpert users with unstable eyes and hands also produce large numbers of digital pictures with annoying artifacts during acquisition. Encountering these reference images with imperfect perceptual quality and their compressed derivatives, FR algorithms may present inferior performance since the majority of them are targeted on high-quality references. Although NR algorithms can make predictions without referring to the source images, few of them are capable of handling both authentic distortions and synthetic compressions effectively. What’s more, existing NR-IQA solutions suffer from many limitations when used in distorted-then-compressed image quality assessment. On the one hand, traditional NR-IQA methods mainly adapt their features to handle simulated distortions, instead of the authentic distortions. Specifically, the perceived quality of distorted-then-compressed images depends not only on the introduced compression but also on the process of photography. For example, the distortion type identification sub-network of MEON [12] tends to be helpless towards complicated authentic distortions. On the other hand, although existing general-purpose NR-IQA methods can be used to evaluate the perceptual quality of distorted-then-compressed images, their predictions are less reliable in terms of consistency with the corresponding subjective records. Since some underlying assumptions of them are flawed (such as the Gaussian assumption in NIQE [13] and GM-LOG [14]), which may not be applicative for the distorted-then-compressed scenario.

To highlight the above problem, a high-quality and a low-quality reference along with their associated compressed versions sampled from the LIVE Wild Compressed Picture Quality Database (LIVE Compressed) [15] are presented in Fig. 1. These images are displayed with their subjective mean opinion scores (MOS) along with the predicted quality scores by several objective IQA models, where we use MS-SSIM [8] (in a range of [0, 1], higher values indicate better quality) as an exemplar FR module, NIQE, and DB-CNN [16] (both in a range of [0, 100], higher values indicate worse quality) as two exemplar NR modules. We observe that the MS-SSIM, NIQE, and DB-CNN values are all in monotonic agreement with MOSs in Fig. 1(a)-(c) or Fig. 1(d)-(e), which indicates that these IQA models deal with this scenario well. Nevertheless, taking a cross-comparison between Fig. 1(d)-(e), and Fig. 1(e), we find the disagreement that a lower MOS corresponds to a higher MS-SSIM score. Indeed, the ground truths strongly indicate that the perceptual quality of Fig. 1(b) is better than that of Fig. 1(e). Similar observations can be captured from Fig. 1(e) and Fig. 1(f), where the NIQE model also becomes unreliable as MS-SSIM. DB-CNN is no exception in Fig. 1(f), and Fig. 1(d). The reason for these inconsistencies is that most existing general-purpose IQA metrics are designed for images with either simulated or authentic distortions, hence it is troublesome for them to reliably predict the perceptual quality of distorted-then-compressed images. This practical dilemma motivates us to seek fundamental solutions to accurately predict the quality of images.

In this paper, we devise an end-to-end learnable framework for this authentically-distorted then synthetically-compressed scenario. On the one hand, we employ a deep residual neural network [17] initialized on ImageNet [18] and further pre-trained on the IQA database KonIQ-10 k [19] to account for authentic distortions, which serves as a good initialization since it has met enormous realistic images with different quality during the transfer learning process. On the other hand, similar to previous work [12], [16], we establish a large-scale synthetic training set based on the Waterloo Exploration database [20] and KADIS-700 k dataset [21], which takes the compression levels into account. We then establish and pre-train a CNN through a multi-class classification task to identify the severity of synthetic compressions. Next, two activations from the authentic-distortion-aware and synthetic-compression-aware branches are bilinearly pooled to obtain the final image representation. Furthermore, when assessing the ultimately compressed images, we fine-tune the overall architecture from an elaborately-designed auxiliary dataset for a cross-database test to demonstrate the robustness of the proposed algorithm. Last but not the least, we combine the products of the authentic-distortion-aware pipeline on the reference image and the overall architecture on the compressed image to predict the quality in a two-step fashion. Assisted with the above-mentioned strategies, extensive experiments on the distorted-then-compressed scenario show standout efficiency by producing more reliable objective prediction scores against state-of-the-art IQA algorithms. The main contributions of this work can be summarized as follows:

• We propose a new stand-alone NR-IQA model for the distorted-then-compressed scenario, without using the corresponding authentically-distorted reference images. It is straightforward to pre-train, and fine-tune the proposed network to accommodate hybrid distortions in this scenario with easily-accessible training data.

• We annotate a large-scale self-collected distorted-then-compressed training set with proxy quality scores, which effectively bypass the expensive and time-consuming subjective testing. With this strategy, our model can be trained and make predictions in an opinion-unaware manner, which significantly improves the generalization of our method.

• We implement the two-step methodology in a unified framework to further boost the performance, which does not rely on any other FR-IQA or NR-IQA metrics.

The remainder of this paper is organized as follows. In Section 2 we give an overview of related metrics including distortion-related NR-IQA methods and the distorted-then-compressed scenario. Section 3 describes the proposed method in detail. Section 4 exhibits the experimental results, corresponding analysis, and comparisons to related algorithms on mainstream IQA databases. Section 5 concludes the paper.