Single image reflection removal based on structure-texture layering

https://doi.org/10.1016/j.image.2020.115868Get rights and content

Highlights

  • This paper propose a unified framework which handles reflection removal and artifact suppression simultaneously.

  • This paper adopts structure-texture decomposition to remove the artifacts that are introduced in the reflection removal operation.

  • The experiments show that our method outperforms several previous methods on both synthetic and real-world images.

Abstract

In computer vision, the reflections impair the overall perception of images and introduce undesired interference to high-level image understanding. To date, reflection removal approaches have been developed with advantage of visual quality enhancement. However, the reflection removal for single image is still a challenging task with its performance to be further boosted. During this process, additional visual artifacts such as blocking may also be introduced that degrades subjective quality of images. In this paper, we make the first attempt to eliminate image reflections and artifacts simultaneously within a unified framework. A structure-texture strategy is utilized to decompose the image into structure and texture layers. The proposed model handles reflection removal in both layers and further suppress blocking artifacts in the texture layer. Finally, a fusion process is employed to obtain the result image. Experiments with objective and subjective evaluations validate the superiority of the proposed method, which outperforms several state-of-the-art algorithms.

Introduction

Current computer vision methods aim to deal with capture images through different mobile devices, and make them gain high level understanding. In real world, images taken through a glass usually contain undesirable reflections. Not only do these reflections damage the image content, they also influence the performance of computer vision tasks such as visual tracking, object detection and video analysis [1], [2], [3], [4], [5], [6], [7]. Therefore, the removal of undesired reflections has become an essential task to obtain high-quality images for both human visual and computer understanding. Mathematically, the observed reflection images can be expressed by the following linear model  [8]: I=IB+IR,where I is the observed image and IB, IR are the background layer and reflection layer, respectively. This layer separation problem is an ill-posed one as the number of unknowns to be recovered is twice as many as that of the input. Solutions to make the problem tractable have to impose constraints based on assumptions from prior knowledge, thus condensing the space of valid solutions. Multiple images make reflection removal less ill-posed and show better performance results than the single image, but the single image reflection removal task is of great attention to researchers due to its practicability for daily life. In this paper, we focus only on single image reflection removal.

The previous existing methods in single image reflection removal depend on gradient sparsity prior. Their capabilities are also limited due to the assumption that reflections are weaker and smoother than the background [1], [9]. A successful approach [2] suppressed the reflection of a single input instead of separating the image into reflection and background layers. The reasons are two fold. First, perfect layer separation of a single image is in general difficult; Second, the separated layers using existing methods contain misclassified information, especially when the reflection is sharp and strong. Experiments in this work [2] show a promising performance (Fig. 5). In most computer vision scenes, human visual system has a strong sensitivity to the background layer of a single image. Also, single reflection removal is in general difficult. Fig. 1 is an example of our proposed method applying on a real-world image. More recently, some methods based on deep learning [4], [5], [10], [11] can achieve desirable results of reflection removal, [5] proposed a generic deep architecture to remove reflection by estimating edges and reconstructing images using only cascaded convolutional layers, and [11] proposed a novel exclusion loss that effectively enforces the separation of transmission and reflection at pixel level. However, these learning-based methods need a deep complex trained networks or extra training time are required. In this work, We attempt to remove reflection via a model-based approach.

We revisit this challenging reflection problem. The real-world images are generally compressed for efficient storage and transmission, which lead to undesired artifacts. Existing single reflection removal methods generally focus on reflection removal without consideration the resulting visual artifacts such as blocking. [2] proposed an image reflection suppression approach that is highly efficient. However, in real world, the existing approaches [3], [9], [12], [13], [14] fail to completely remove sharp and strong reflection for compressed images. [15] proposed a framework based on structure-texture decomposition to remove the compression artifacts that are amplified in the image contrast enhancement operation. Our idea is inspired by [15], [16], We attempt to remove the compression artifacts and reflection in the image restoration operation, respectively. In this paper, we first propose a novel framework of image reflection (Fig. 2), which is based on structure-texture and handles reflections of single image effectively. Meanwhile, it provides promising results compared to the existing the state-of-the-art model-based methods in single image reflection removal on both real (Fig. 7) and synthetic images (Fig. 5). Our main contributions are as follows:

  • (1)

    We first propose a unified framework which handles reflection removal and artifact suppression simultaneously.

  • (2)

    Our framework utilizes structure-texture decomposition to remove the additional artifacts that are introduced in the image reflection removal operation.

  • (3)

    We show better results on synthetic and real-world images with respect to previous approaches in single-image reflection removal.

The remainder of this paper is organized as follows. Section 2 discusses the related methods that deal with image reflection, including multiple image reflection removal and single image removal. Section 3 details our proposed framework, including the problem formulation and image processing, optimization. Section 4 shows the results and analysis of experiments on both real and synthetic images in comparison with results of other methods. Finally, the paper is concluded in Section 5.

Section snippets

Related work

There are a number of methods proposed for image reflection removal. These fall into two categories: multiple-image reflection removal and single-image reflection removal.

Proposed method

Our framework is shown in Fig. 2. The original input image is decomposed into two layers: structure and texture layers. Formally, the model can be expressed as: I=IS+IT,where IS, IT are the structure and texture layer, respectively. In this paper, we handle both layers in different ways. The texture layer is processed by reflection removal, then a combination of image mask remove blocking artifact. While in structure layer, we first eliminate both reflection and blocking artifact and then

Experiments

Our experiments are conducted on a PC with 8-core Intel i5-8520U 1.80 GHz CPU and 8 GB RAM. The implementation is done using MATLAB2019a without any GPU acceleration. We evaluate our method on both synthetic and real-world images, and compare our method with the state-of-the-art methods, the results can be visualized in Fig. 5, Fig. 7. In addition, we also test our method on a recent public datasets [6]. We choose state-of-the-art methods [1], [2], [3], [12] as comparisons. For the experiments

Conclusion

In this paper, we propose a framework to remove reflections and artifacts in single image simultaneously. The proposed framework decomposes the input image into structure and texture layers with this unified framework. The advantage is that we can remove reflection and blocking artifact simultaneously while retaining more details. We validate the effectiveness of our method through experiments on synthetic, real-world images and public datasets. Our proposed method outperforms recent

CRediT authorship contribution statement

Nanfeng Jiang: Writing original draft, Methodology, Data curation. Yuzhen Niu: Validation. Weiling Chen: Resources, Funding acquisition. Liqun Lin: Visualization. Nadir Mustafa: Validation. Tiesong Zhao: Supervision, Writing - review & editing, Funding acquisition, Project administration.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This research is supported by the National Natural Science Foundation of China (Grants 61671152, 61901119).

References (29)

  • XuL. et al.

    Structure extraction from texture via relative total variation

    ACM Trans. Graph.

    (2012)
  • Y. Li, M.S. Brown, Single image layer separation using relative smoothness, in: Proceedings of the IEEE Conference on...
  • N. Arvanitopoulos, R. Achanta, S. Susstrunk, Single image reflection suppression, in: Proceedings of the IEEE...
  • Y. Yang, W. Ma, Y. Zheng, J.-F. Cai, W. Xu, Fast single image reflection suppression via convex optimization, in:...
  • J. Yang, D. Gong, L. Liu, Q. Shi, Seeing deeply and bidirectionally: A deep learning approach for single image...
  • Q. Fan, J. Yang, G. Hua, B. Chen, D. Wipf, A generic deep architecture for single image reflection removal and image...
  • R. Wan, B. Shi, L.-Y. Duan, A.-H. Tan, A.C. Kot, Benchmarking single-image reflection removal algorithms, in:...
  • C. Wang, B. Shi, L. Duan, Learning to remove reflections from windshield images, 78 (2019) 94–102....
  • BarrowH. et al.

    Recovering intrinsic scene characteristics

    Comput. Vis. Syst.

    (1978)
  • LevinA. et al.

    User assisted separation of reflections from a single image using a sparsity prior

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2007)
  • R. Wan, B. Shi, L.-Y. Duan, A.-H. Tan, A.C. Kot, Crrn: Multi-scale guided concurrent reflection removal network, in:...
  • X. Zhang, R. Ng, Q. Chen, Single image reflection separation with perceptual losses, in: Proceedings of the IEEE...
  • WanR. et al.

    Depth of field guided reflection removal

  • SpringerO. et al.

    Reflection separation using guided annotation

  • View full text