Nighttime image dehazing based on Retinex and dark channel prior using Taylor series expansion
Introduction
Images captured outdoors in fog and haze weather engender the degeneracy phenomenon, such as low contrast or color distortion, which seriously affects the performance of outdoor visual systems. According to Middleton’s model (Middleton, 1952), a haze image can be described as a linear combination of a direct attenuation term and a scattering term as: where is a pixel coordinate, is the observed haze image, is its haze-free image, is the global atmospheric light, is a scene albedo, and is the transmission map depending on the scattering coefficient and scene depth . To retrieve , and need to be estimated, which lead to an underdetermined estimation problem.
Based on Middleton’s model (Middleton, 1952), a number of methods have been proposed to remove haze from single images (He et al., 2011, Meng et al., 2013, Fattal, 2014, Wang et al., 2015, Zhu et al., 2015, Berman et al., 2016, Jiang et al., 2017, Wang et al., 2017, Zhu et al., 2017, Galdran et al., 2018, Ren et al., 2020). The key to their success is based on the above optical model in Eq. (1) and various image priors, e.g., the dark channel prior in He et al. (2011) and the color attenuation prior in Zhu et al. (2015). Although these methods are effective on dehazing daytime images, they are not effective enough when dealing with nighttime haze images. Fig. 1 shows the dehazing results on a nighttime haze image using different methods. The daytime dehazing method (He et al., 2011) has failed to remove haze (see Fig. 1(b)), and the nighttime dehazing method (Zhang et al., 2017) is erroneous in dealing with glow and produces color distortion(see Fig. 1(c)). As shown in Fig. 1(d), our proposed method has obtained good and more natural dehazing effect. The main reason is that the above daytime haze imaging model (Middleton, 1952) does not actually suit the nighttime haze scenes (Zhang et al., 2017) and some priors cannot be established either. For instance, the dark channel prior in He et al. (2011) does not hold for the illuminating conditions of nighttime haze scenes because of multiple artificial light sources (see Fig. 1(a)).
Recently, some solutions for nighttime haze removal were emerged (Zhang et al., 2017, Zhang et al., 2014, Li et al., 2015, Liao et al., 2018, Yang et al., 2018). In Zhang et al. (2017), a novel maximum reflectance prior was proposed to process single nighttime haze images. The prior came from a key observation that the local maximum intensities were mainly contributed by the ambient illumination. Using maximum reflectance prior, the ambient illumination and transmission map were estimated to restore a haze-free image. The algorithm may fail to dispose some cases with color distortion such as the regions of grasses or leaves. Zhang et al. (2014) proposed a new nighttime dehazing method with illumination estimation. The method, which utilized the Retinex algorithm to estimate and enhance the incident light intensity, adopted the gamma adjustment for color correction, and used the dark channel prior to obtain a haze-free image. Because of the additional post processing, it may result in color distortion. Li et al. (2015) introduced a new nighttime haze model, which accounted for the varying light sources and their glow. The model regarded the glow as an atmospheric point spread function, and used the layer separation technique in Li and Brown (2014) to remove the glow. The method can effectively eliminate the halo artifacts, but the output image appears to be unnatural and is prone to over-enhancing some local areas. Liao et al. (2018) proposed an end-to-end learning-based solution to remove haze from nighttime images. The solution used a novel model to represent a nighttime haze image, and adopted a Haze Density Prediction Network (HDP-Net) to obtain a haze density map, which was used to generate a haze-free output. However, the dehazing effect of output image is not very obvious and is liable to produce halo effect. Yang et al. (2018) proposed a novel superpixel-based single image haze removal algorithm for nighttime haze images. Their algorithm utilized the superpixel-based method to compute the value of the atmospheric light and dark channel in the glow-free haze image, which was decomposed from an input nighttime image. Then, the dark channel was used to estimate the transmission map, and an adaptive threshold was added to the transmission map for obtaining a restored image. The algorithm has achieved good results in image haze removal, but it may generate exaggerated intensity of some areas such as artificial light sources and produce color distortion.
In this paper, following the Dark channel prior we propose a simple but effective nighttime haze removal method based on the Retinex theory and Taylor series expansion. For the ease of reference, we refer to this approach as ‘RDT’ dehazing. As we all know, Retinex (Galdran et al., 2018, Vazquez-Corral and Finlayson, 2019) is a color vision model mimicking the ability of the Human Visual System to robustly discount varying illuminations when observing a scene under different spectral lighting conditions. In Zhang et al. (2014), the Retinex algorithm was used to enhance image brightness. Different from the method proposed in Zhang et al. (2014), our proposed RDT method adopts the Retinex theory to first decompose the atmospheric light from the input image and then uses it to accurately estimate the transmission map so as to dehaze nighttime images. Similar to the methods in Zhang et al. (2014) and Li et al. (2015), our proposed RDT method utilizes the dark channel prior to realize the haze removal process. However, to obtain a more accurate pointwise estimation of the transmission map for nighttime haze images, we for the first time introduce the Taylor series expansion in this process. Finally, to further improve the quality of the restored image both in the aspects of illumination and color, image fusion (Li et al., 2013, Galdran et al., 2016, Deng et al., 2019, Guo et al., 2020) and color transfer (Reinhard et al., 2001) algorithms are adopted in our approach. The quantitative and qualitative comparison with several recent state-of-the-art methods demonstrates that our proposed RDT method is robust and effective.
The remainder of this paper is organized as follows. In Section 2, the nighttime haze imaging model is briefly introduced. In Section 3, our proposed method is illustrated in details. In Section 4, experiments and results are presented and analyzed. Finally, the conclusion is drawn in Section 5.
Section snippets
Nighttime haze imaging model
Eq. (1) gives the daytime haze imaging model, but it is inappropriate to describe nighttime haze images due to the presence of artificial light sources. For example, MSCNN-HE (Ren et al., 2020) was trained on a synthetic dataset created with a daytime haze imaging model. Therefore, it is less effective for nighttime hazy images or images with non-uniform atmospheric light, where it results in some dark regions due to the inaccuracy of the estimated atmospheric light.
Recently, several nighttime
The proposed method
As shown in the flow chart in Fig. 2, our proposed method consists of the following three parts and are detailed in the subsections below. (1) Atmospheric Light Decomposition. The shades-of-gray color correction algorithm (Finlayson and Trezzi, 2004) is first adopted to deal with the nighttime haze image because of the colored artificial light sources, and then Retinex theory is performed on the color-corrected haze image to decompose the atmospheric light image. (2) Transmission Map Estimation
Experiments and results
To demonstrate the performance of the proposed method, a series of experiments are conducted to compare with several state-of-the-art methods (He et al., 2011, Zhang et al., 2017, Zhang et al., 2014, Li et al., 2015, Liao et al., 2018, Yang et al., 2018) on synthetic and natural haze images. All of the state-of-the-art results are obtained using the original codes provided by the corresponding authors on their homepages or GitHub webpages. We evaluate and compare the dehazing results in both
Conclusion
In this paper, a novel method of using a Taylor series expansion based on Retinex and dark channel prior has been proposed for nighttime image dehazing. Compared with the state-of-the-art methods, the proposed method can effectively correct color distortion, improve image clarity and generate more pleasant image details. This work has further exploited the advantages of dark channel prior using the Taylor series expansion for nighttime haze images. In addition, from Table 1, more assessment
CRediT authorship contribution statement
Qunfang Tang: Conceived the presented idea, Developed the theory and performed the computations, Drafted the article, Discussed the results and contributed to the final manuscript. Jie Yang: Conceived the presented idea, Critical revision of the article, Discussed the results and contributed to the final manuscript. Xiangjian He: Conceived the presented idea, Critical revision of the article, Discussed the results and contributed to the final manuscript. Wenjing Jia: Developed the theory and
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to thank the authors of compared papers, who provided related images and original codes, and also the anonymous reviewers for their insightful comments and valuable suggestions. The authors gratefully acknowledge financial support from China Scholarship Council.
This work was supported by National Natural Science Foundation of China (No.51879211), Hunan Provincial Natural Science Foundation of China (No.2017JJ3053), Hunan Provincial Key Research and Development Project of
References (49)
- et al.
Single image dehazing based on fusion strategy
Neurocomputing
(2020) - et al.
Image dehazing using adaptive bi-channel priors on superpixels
Comput. Vis. Image Underst.
(2017) - et al.
Single image dehazing with a physical model and dark channel prior
Neurocomputing
(2015) - et al.
Atmospheric light estimation in hazy images based on color-plane model
Comput. Vis. Image Underst.
(2017) - Afifi, M., Brown, M.S., 2019. What else can fool deep learning? Addressing color constancy errors on deep neural...
- Afifi, M., Price, B., Cohen, S., Brown, M.S., 2019. When color constancy goes wrong: Correcting improperly...
- Akkaynak, D., Treibitz, T., 2019. Sea-thru: A method for removing water from underwater images. In: Proceedings of the...
- et al.
Color balance and fusion for underwater image enhancement
IEEE Trans. Image Process.
(2017) - et al.
Color balance and fusion for underwater image enhancement
IEEE Trans. Image Process.
(2018) - Anon, .,...
Dehazenet: An end-to-end system for single image haze removal
IEEE Trans. Image Process.
Underwater image enhancement by wavelength compensation and dehazing (wcid)
IEEE Trans. Image Process.
Color image enhancement based on single-scale retinex with a jnd-based nonlinear filter
Referenceless prediction of perceptual fog density and perceptual image defogging
IEEE Trans. Image Process.
Dehazing using color-lines
ACM Trans. Graph. (TOG)
Shades of Gray and Colour Constancy
Retinex in matlab
J. Electron. Imaging
Fusion-based variational image dehazing
IEEE Signal Process. Lett.
Single image haze removal using dark channel prior
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (37)
Revisiting coarse-to-fine strategy for low-light image enhancement with deep decomposition guided training
2024, Computer Vision and Image UnderstandingRecent advances in image dehazing: Formal analysis to automated approaches
2024, Information FusionEfficient image dehazing algorithm using multiple priors constraints
2023, Journal of Visual Communication and Image RepresentationCitation Excerpt :These algorithms consider the contrast reduction characteristics in hazy images and emphasise the enhancement in texture information in low-visibility images. Representative methods include histogram equalisation (HE) [3], retinex [4], wavelet transform [5], and homomorphic filtering [6]. However, these algorithms ignore the specific image-quality degradation mechanism, which limits the quality of the final output images.
Rapid nighttime haze removal with color-gray layer decomposition
2022, Signal ProcessingCitation Excerpt :Notably, the reason why these approaches [20–22,24] can cause massive color artifacts is that color constraint in the glow-haze decomposition [20] fails for nighttime hazy scenes with severe color attenuation. Tang et al. [25] proposed an approach based on Retinex theory and Taylor series expansion to implement nighttime image dehazing. Recently, the learning-based methods [26–28] have made significant progress in nighttime haze removal.
Visibility enhancement and dehazing: Research contribution challenges and direction
2022, Computer Science ReviewCitation Excerpt :RESIDE-SOTS, NYU-depth, Dense Haze, I-Haze, O-Haze, and SUN-Haze [219] are grouped as synthetic image dataset and RESIDE-RTTS, RESIDE-HSTS, Night-haze [220], some frequently used real-world image sets and our Dense to Low (D2L-Haze) real-world hazy image set are categorized into real-world image dataset. We choose 29 dense and 55 varicolored images from RESIDE-RTTS [158] dataset and 10 frequently used night and halation-effect images from SUN-HAZE [219] and night datasets [220]. . As dehazing approaches are sensitive to the sky and white regions, six real-world images are selected that are hard to handle and challenging for the evaluation of the dehazing approaches.
- 1
Senior Member, IEEE.