Single image reflection removal based on structure-texture layering
Introduction
Current computer vision methods aim to deal with capture images through different mobile devices, and make them gain high level understanding. In real world, images taken through a glass usually contain undesirable reflections. Not only do these reflections damage the image content, they also influence the performance of computer vision tasks such as visual tracking, object detection and video analysis [1], [2], [3], [4], [5], [6], [7]. Therefore, the removal of undesired reflections has become an essential task to obtain high-quality images for both human visual and computer understanding. Mathematically, the observed reflection images can be expressed by the following linear model [8]: where is the observed image and , are the background layer and reflection layer, respectively. This layer separation problem is an ill-posed one as the number of unknowns to be recovered is twice as many as that of the input. Solutions to make the problem tractable have to impose constraints based on assumptions from prior knowledge, thus condensing the space of valid solutions. Multiple images make reflection removal less ill-posed and show better performance results than the single image, but the single image reflection removal task is of great attention to researchers due to its practicability for daily life. In this paper, we focus only on single image reflection removal.
The previous existing methods in single image reflection removal depend on gradient sparsity prior. Their capabilities are also limited due to the assumption that reflections are weaker and smoother than the background [1], [9]. A successful approach [2] suppressed the reflection of a single input instead of separating the image into reflection and background layers. The reasons are two fold. First, perfect layer separation of a single image is in general difficult; Second, the separated layers using existing methods contain misclassified information, especially when the reflection is sharp and strong. Experiments in this work [2] show a promising performance (Fig. 5). In most computer vision scenes, human visual system has a strong sensitivity to the background layer of a single image. Also, single reflection removal is in general difficult. Fig. 1 is an example of our proposed method applying on a real-world image. More recently, some methods based on deep learning [4], [5], [10], [11] can achieve desirable results of reflection removal, [5] proposed a generic deep architecture to remove reflection by estimating edges and reconstructing images using only cascaded convolutional layers, and [11] proposed a novel exclusion loss that effectively enforces the separation of transmission and reflection at pixel level. However, these learning-based methods need a deep complex trained networks or extra training time are required. In this work, We attempt to remove reflection via a model-based approach.
We revisit this challenging reflection problem. The real-world images are generally compressed for efficient storage and transmission, which lead to undesired artifacts. Existing single reflection removal methods generally focus on reflection removal without consideration the resulting visual artifacts such as blocking. [2] proposed an image reflection suppression approach that is highly efficient. However, in real world, the existing approaches [3], [9], [12], [13], [14] fail to completely remove sharp and strong reflection for compressed images. [15] proposed a framework based on structure-texture decomposition to remove the compression artifacts that are amplified in the image contrast enhancement operation. Our idea is inspired by [15], [16], We attempt to remove the compression artifacts and reflection in the image restoration operation, respectively. In this paper, we first propose a novel framework of image reflection (Fig. 2), which is based on structure-texture and handles reflections of single image effectively. Meanwhile, it provides promising results compared to the existing the state-of-the-art model-based methods in single image reflection removal on both real (Fig. 7) and synthetic images (Fig. 5). Our main contributions are as follows:
- (1)
We first propose a unified framework which handles reflection removal and artifact suppression simultaneously.
- (2)
Our framework utilizes structure-texture decomposition to remove the additional artifacts that are introduced in the image reflection removal operation.
- (3)
We show better results on synthetic and real-world images with respect to previous approaches in single-image reflection removal.
The remainder of this paper is organized as follows. Section 2 discusses the related methods that deal with image reflection, including multiple image reflection removal and single image removal. Section 3 details our proposed framework, including the problem formulation and image processing, optimization. Section 4 shows the results and analysis of experiments on both real and synthetic images in comparison with results of other methods. Finally, the paper is concluded in Section 5.
Section snippets
Related work
There are a number of methods proposed for image reflection removal. These fall into two categories: multiple-image reflection removal and single-image reflection removal.
Proposed method
Our framework is shown in Fig. 2. The original input image is decomposed into two layers: structure and texture layers. Formally, the model can be expressed as: where , are the structure and texture layer, respectively. In this paper, we handle both layers in different ways. The texture layer is processed by reflection removal, then a combination of image mask remove blocking artifact. While in structure layer, we first eliminate both reflection and blocking artifact and then
Experiments
Our experiments are conducted on a PC with 8-core Intel i5-8520U 1.80 GHz CPU and 8 GB RAM. The implementation is done using MATLAB2019a without any GPU acceleration. We evaluate our method on both synthetic and real-world images, and compare our method with the state-of-the-art methods, the results can be visualized in Fig. 5, Fig. 7. In addition, we also test our method on a recent public datasets [6]. We choose state-of-the-art methods [1], [2], [3], [12] as comparisons. For the experiments
Conclusion
In this paper, we propose a framework to remove reflections and artifacts in single image simultaneously. The proposed framework decomposes the input image into structure and texture layers with this unified framework. The advantage is that we can remove reflection and blocking artifact simultaneously while retaining more details. We validate the effectiveness of our method through experiments on synthetic, real-world images and public datasets. Our proposed method outperforms recent
CRediT authorship contribution statement
Nanfeng Jiang: Writing original draft, Methodology, Data curation. Yuzhen Niu: Validation. Weiling Chen: Resources, Funding acquisition. Liqun Lin: Visualization. Nadir Mustafa: Validation. Tiesong Zhao: Supervision, Writing - review & editing, Funding acquisition, Project administration.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
This research is supported by the National Natural Science Foundation of China (Grants 61671152, 61901119).
References (29)
- et al.
Structure extraction from texture via relative total variation
ACM Trans. Graph.
(2012) - Y. Li, M.S. Brown, Single image layer separation using relative smoothness, in: Proceedings of the IEEE Conference on...
- N. Arvanitopoulos, R. Achanta, S. Susstrunk, Single image reflection suppression, in: Proceedings of the IEEE...
- Y. Yang, W. Ma, Y. Zheng, J.-F. Cai, W. Xu, Fast single image reflection suppression via convex optimization, in:...
- J. Yang, D. Gong, L. Liu, Q. Shi, Seeing deeply and bidirectionally: A deep learning approach for single image...
- Q. Fan, J. Yang, G. Hua, B. Chen, D. Wipf, A generic deep architecture for single image reflection removal and image...
- R. Wan, B. Shi, L.-Y. Duan, A.-H. Tan, A.C. Kot, Benchmarking single-image reflection removal algorithms, in:...
- C. Wang, B. Shi, L. Duan, Learning to remove reflections from windshield images, 78 (2019) 94–102....
- et al.
Recovering intrinsic scene characteristics
Comput. Vis. Syst.
(1978) - et al.
User assisted separation of reflections from a single image using a sparsity prior
IEEE Trans. Pattern Anal. Mach. Intell.
(2007)
Depth of field guided reflection removal
Reflection separation using guided annotation
Cited by (4)
An Adaptive Low-Light Image Enhancement Using Canonical Correlation Analysis
2023, IEEE Transactions on Industrial InformaticsImage Specular Highlight Removal using Generative Adversarial Network and Enhanced Grey Wolf Optimization Technique
2023, International Journal of Advanced Computer Science and ApplicationsRegistration and Fusion Using Reflection in Passive Millimeter-Wave Images
2023, IEEE Transactions on Geoscience and Remote SensingLow-light image enhancement based on exponential Retinex variational model
2021, IET Image Processing