Single nighttime image dehazing based on image decomposition

Highlights

• A novel framework is proposed to simultaneously address major degraded problems of nighttime image dehazing.

• A new structure-texture-noise (STN) decomposition model considering the structure and texture awareness is presented.

• The proposed nighttime dehazing framework can also solve restoration problems of daytime hazy images and low-light images.

Abstract

Dehazing plays an important role in promoting the performance of outdoor computer vision systems. However, existing dehazing methods are targeted to daytime haze scenes, and are not able to improve visual effects for nighttime hazy images due to the unpredictable factors at night. In this paper, an effective single image dehazing framework based on image decomposition is presented for nighttime hazy images. First, the input single nighttime image is separated into the glow-shaped image and the glow-free nighttime hazy image using its relative smoothness constraint. Then, a novel structure-texture-noise decomposition model based on the exponentiated mean local variance is devised to split the nighttime hazy image into a structure layer, a texture layer and a noise layer, in which the structure layer and the texture layer are dehazed based on the maximum reflectance prior and the dark channel prior and enhanced in the gradient domain respectively. Finally, the dehazed structure layer and the enhanced texture layer are fused to produce a dehazed result. Experiments demonstrate that the proposed approach outperforms several state-of-the-art dehazing techniques for nighttime hazy scenes, especially in terms of noise suppression. Besides, the proposed algorithm is also capable of handling daytime hazy images and low-light degraded images.

Introduction

Images or videos acquired under hazy scenes at night are usually plagued by blurring, contrast reduction and color degradation owing to the presence of tiny suspended particles in the atmosphere. These deficiencies will adversely degrade the performance of subsequent vision-based tasks that require high-quality inputs. Therefore, haze removal is highly valuable for outdoor computer vision applications, such as surveillance, intelligent transportation and remote sensing.

To the best of our knowledge, daytime dehazing approaches have been researched widely and great achievements have been made. Earlier dehazing methods mainly make use of multiple images [1], [2] or additional information [3] to restore the hazy image. Unfortunately, these algorithms are generally limited in real applications. Therefore, more and more researchers focus on haze removal from a single image. From the viewpoint of image enhancement, many efforts have been devoted to stress the contrast and improve the colors, such as gray-level remapping [4], fusion-based strategy [5], [6], [7], Retinex-based filtering [8], [9], gradient domain operation [10], [11] and so on. In recent years, another kind of single image dehazing methods is on the basis of the classic haze imaging model, which is first presented by Koschmieder [12] and further derived by Narasimhan and Nayar [13]. Mathematically, the atmospheric scattering model can be written as:

$$I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right)$$

where $x$ is the pixel coordinate within the image, $I\left(x\right)$ and $J\left(x\right)$ stand for the observed hazy image and the restored haze-free image, respectively. $A$ is the global atmospheric light, which is normally assumed as a constant under daytime hazy scenes. The medium transmission $t\left(x\right)$ represents the relative portion of the light received by the camera from the scene radiance without being scattered. The first term $J\left(x\right)t\left(x\right)$ is the direct decay, indicating the attenuation of the reflected light in the medium. The second part $A\left(1-t\left(x\right)\right)$ denotes the airlight shifted component. The core idea of these physics-based dehazing methods depends on various priors or assumptions [14], [15], [16], [17], [18], [19] to estimate two important parameters: the transmission map $t\left(x\right)$ and the global atmospheric light $A$. Subsequently, with the appearance and continuous development of the Graphics Processing Unit (GPU), deep learning models based haze removal algorithms gradually become the focus. Numerous interesting learning frameworks [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31] have been developed to handle the dehazing task by extracting effective haze-relevant features automatically. Although these above dehazing algorithms generally work well for daytime hazy images, most of them are incapable of dealing with nighttime hazy scenes due to the undesirable factors at night such as glow effects, low overall brightness and non-uniform illumination from artificial light sources. As shown in Fig. 1, some advanced daytime dehazing methods cannot improve the quality of the nighttime hazy image. This is mainly because that the traditional atmospheric scattering model cannot explain the formation of the nighttime hazy image.

To address this problem, researchers have started to explore nighttime haze removal. However, as far as we know, there are fewer literatures on nighttime image dehazing within the last decade. Taking the color transfer as a pre-processing step, some works [32], [33], [34] first map the colors of a nighttime hazy image onto those of a daytime hazy image, and then attempt to adopt generic dehazing or enhancement strategies to process it. Influencing by the color transfer, such approaches may lead to unnatural haze-free results. Some new models and physics-based methods [35], [36], [37], [38], [39] are presented to describe the degradation features (e.g. non-uniform illumination and glow effects) of the nighttime hazy image. Subsequently, fusion-based strategy [40], [41], [42], [43] are used to remove the nighttime haze and alleviate the glow effects. Recently, learning-based dehazing frameworks [44], [45], [46] are also devised for nighttime hazy images. However, these existing models fail to consider the noise of the nighttime hazy image and most existing dehazing methods generally suffer from noise amplification.

To overcome the above limitations, inspired by image decomposition, an effective dehazing algorithm is proposed for nighttime hazy images. On the basis of the observation that the degradation of a nighttime hazy image mainly includes glow effects, haze interference, texture blurring and noise amplification, we first define a model by using a linear combination of four terms: the glow term, the structure term, the texture term, and the noise term. Based on this linear model, the input single nighttime hazy image is first decomposed into four image layers: a glow layer, a structure layer, a texture layer and a noise layer. Next, the dehazing operation based on the dark channel prior and the maximum reflection prior is only employed across the residual structure layer rather than the entire image because the haze generally affects the low frequency of image. Then, the contrast of the texture layer will be enhanced in the gradient domain. Finally, we integrate the dehazed structure layer and the enhanced texture layer to generate the dehazing result. The proposed nighttime dehazing methods can not only guarantee the dehazing ability and texture enhancement, but also suppress the unwanted noise. More specifically, the main contributions are as follows:

• A novel framework is proposed to simultaneously address four major degraded problems of nighttime image dehazing: glow effects, haze interference, texture blurring and noise amplification.

• A new structure-texture-noise (STN) decomposition model considering the structure and texture awareness is presented to simultaneously separate the worthless noise layer and obtain the structure and texture layer, which can suppress the noise and improve the visual effects of structure and texture information at the same time.

• The proposed nighttime dehazing framework can also solve the restoration problem of other degraded images such as daytime hazy images and low-light images.

The remainder of this paper is organized as follows. Section 2 briefly reviews the related work, including daytime dehazing methods and nighttime dehazing methods. The details of our proposed nighttime dehazing framework is given in Section 3. Section 4 shows the experimental results and analysis. Finally, the conclusion and discussion are introduced in Section 5.