Nighttime image dehazing based on Retinex and dark channel prior using Taylor series expansion

https://doi.org/10.1016/j.cviu.2020.103086Get rights and content

Highlights

  • Giving an effective nighttime dehazing method based on Retinex and Dark channel prior.

  • Proposing color transfer to handle various artificial light sources and their glow.

  • Introducing Taylor series expansion to estimate the pointwise transmission map.

Abstract

Haze removal from nighttime images is more difficult compared with daytime image dehazing due to the uneven illumination, low contrast and severe color distortion. In this paper, following the approaches based on Dark channel prior, we propose a simple yet effective approach using Retinex theory and Taylor series expansion for nighttime image dehazing, referred to as ‘RDT’. Existing nighttime image dehazing methods do not handle color shift and glow removal very well. In order to address these issues, we first propose to decompose the atmospheric light image from the input image based on the Retinex theory. Taylor series expansion is then introduced for the first time to accurately estimate the pointwise transmission map. Finally, during the following processes of image fusion and color transfer, the atmospheric light image and potential haze-free image are adopted to obtain the final haze-free image. The experimental results on benchmark nighttime haze images demonstrate the superior performance of our proposed RDT dehazing method over the state-of-the-art methods.

Introduction

Images captured outdoors in fog and haze weather engender the degeneracy phenomenon, such as low contrast or color distortion, which seriously affects the performance of outdoor visual systems. According to Middleton’s model (Middleton, 1952), a haze image I(x) can be described as a linear combination of a direct attenuation term and a scattering term as: I(x)=J(x)t(x)+A[1t(x)]=Aρ(x)t(x)+A[1t(x)],where x is a pixel coordinate, I(x) is the observed haze image, J(x) is its haze-free image, A is the global atmospheric light, ρ(x) is a scene albedo, and t(x)=eβd(x) is the transmission map depending on the scattering coefficient β and scene depth d(x). To retrieve J(x), A and t(x) need to be estimated, which lead to an underdetermined estimation problem.

Based on Middleton’s model (Middleton, 1952), a number of methods have been proposed to remove haze from single images (He et al., 2011, Meng et al., 2013, Fattal, 2014, Wang et al., 2015, Zhu et al., 2015, Berman et al., 2016, Jiang et al., 2017, Wang et al., 2017, Zhu et al., 2017, Galdran et al., 2018, Ren et al., 2020). The key to their success is based on the above optical model in Eq. (1) and various image priors, e.g., the dark channel prior in He et al. (2011) and the color attenuation prior in Zhu et al. (2015). Although these methods are effective on dehazing daytime images, they are not effective enough when dealing with nighttime haze images. Fig. 1 shows the dehazing results on a nighttime haze image using different methods. The daytime dehazing method (He et al., 2011) has failed to remove haze (see Fig. 1(b)), and the nighttime dehazing method (Zhang et al., 2017) is erroneous in dealing with glow and produces color distortion(see Fig. 1(c)). As shown in Fig. 1(d), our proposed method has obtained good and more natural dehazing effect. The main reason is that the above daytime haze imaging model (Middleton, 1952) does not actually suit the nighttime haze scenes (Zhang et al., 2017) and some priors cannot be established either. For instance, the dark channel prior in He et al. (2011) does not hold for the illuminating conditions of nighttime haze scenes because of multiple artificial light sources (see Fig. 1(a)).

Recently, some solutions for nighttime haze removal were emerged (Zhang et al., 2017, Zhang et al., 2014, Li et al., 2015, Liao et al., 2018, Yang et al., 2018). In Zhang et al. (2017), a novel maximum reflectance prior was proposed to process single nighttime haze images. The prior came from a key observation that the local maximum intensities were mainly contributed by the ambient illumination. Using maximum reflectance prior, the ambient illumination and transmission map were estimated to restore a haze-free image. The algorithm may fail to dispose some cases with color distortion such as the regions of grasses or leaves. Zhang et al. (2014) proposed a new nighttime dehazing method with illumination estimation. The method, which utilized the Retinex algorithm to estimate and enhance the incident light intensity, adopted the gamma adjustment for color correction, and used the dark channel prior to obtain a haze-free image. Because of the additional post processing, it may result in color distortion. Li et al. (2015) introduced a new nighttime haze model, which accounted for the varying light sources and their glow. The model regarded the glow as an atmospheric point spread function, and used the layer separation technique in Li and Brown (2014) to remove the glow. The method can effectively eliminate the halo artifacts, but the output image appears to be unnatural and is prone to over-enhancing some local areas. Liao et al. (2018) proposed an end-to-end learning-based solution to remove haze from nighttime images. The solution used a novel model to represent a nighttime haze image, and adopted a Haze Density Prediction Network (HDP-Net) to obtain a haze density map, which was used to generate a haze-free output. However, the dehazing effect of output image is not very obvious and is liable to produce halo effect. Yang et al. (2018) proposed a novel superpixel-based single image haze removal algorithm for nighttime haze images. Their algorithm utilized the superpixel-based method to compute the value of the atmospheric light and dark channel in the glow-free haze image, which was decomposed from an input nighttime image. Then, the dark channel was used to estimate the transmission map, and an adaptive threshold was added to the transmission map for obtaining a restored image. The algorithm has achieved good results in image haze removal, but it may generate exaggerated intensity of some areas such as artificial light sources and produce color distortion.

In this paper, following the Dark channel prior we propose a simple but effective nighttime haze removal method based on the Retinex theory and Taylor series expansion. For the ease of reference, we refer to this approach as ‘RDT’ dehazing. As we all know, Retinex (Galdran et al., 2018, Vazquez-Corral and Finlayson, 2019) is a color vision model mimicking the ability of the Human Visual System to robustly discount varying illuminations when observing a scene under different spectral lighting conditions. In Zhang et al. (2014), the Retinex algorithm was used to enhance image brightness. Different from the method proposed in Zhang et al. (2014), our proposed RDT method adopts the Retinex theory to first decompose the atmospheric light from the input image and then uses it to accurately estimate the transmission map so as to dehaze nighttime images. Similar to the methods in Zhang et al. (2014) and Li et al. (2015), our proposed RDT method utilizes the dark channel prior to realize the haze removal process. However, to obtain a more accurate pointwise estimation of the transmission map for nighttime haze images, we for the first time introduce the Taylor series expansion in this process. Finally, to further improve the quality of the restored image both in the aspects of illumination and color, image fusion (Li et al., 2013, Galdran et al., 2016, Deng et al., 2019, Guo et al., 2020) and color transfer (Reinhard et al., 2001) algorithms are adopted in our approach. The quantitative and qualitative comparison with several recent state-of-the-art methods demonstrates that our proposed RDT method is robust and effective.

The remainder of this paper is organized as follows. In Section 2, the nighttime haze imaging model is briefly introduced. In Section 3, our proposed method is illustrated in details. In Section 4, experiments and results are presented and analyzed. Finally, the conclusion is drawn in Section 5.

Section snippets

Nighttime haze imaging model

Eq. (1) gives the daytime haze imaging model, but it is inappropriate to describe nighttime haze images due to the presence of artificial light sources. For example, MSCNN-HE (Ren et al., 2020) was trained on a synthetic dataset created with a daytime haze imaging model. Therefore, it is less effective for nighttime hazy images or images with non-uniform atmospheric light, where it results in some dark regions due to the inaccuracy of the estimated atmospheric light.

Recently, several nighttime

The proposed method

As shown in the flow chart in Fig. 2, our proposed method consists of the following three parts and are detailed in the subsections below. (1) Atmospheric Light Decomposition. The shades-of-gray color correction algorithm (Finlayson and Trezzi, 2004) is first adopted to deal with the nighttime haze image because of the colored artificial light sources, and then Retinex theory is performed on the color-corrected haze image to decompose the atmospheric light image. (2) Transmission Map Estimation

Experiments and results

To demonstrate the performance of the proposed method, a series of experiments are conducted to compare with several state-of-the-art methods (He et al., 2011, Zhang et al., 2017, Zhang et al., 2014, Li et al., 2015, Liao et al., 2018, Yang et al., 2018) on synthetic and natural haze images. All of the state-of-the-art results are obtained using the original codes provided by the corresponding authors on their homepages or GitHub webpages. We evaluate and compare the dehazing results in both

Conclusion

In this paper, a novel method of using a Taylor series expansion based on Retinex and dark channel prior has been proposed for nighttime image dehazing. Compared with the state-of-the-art methods, the proposed method can effectively correct color distortion, improve image clarity and generate more pleasant image details. This work has further exploited the advantages of dark channel prior using the Taylor series expansion for nighttime haze images. In addition, from Table 1, more assessment

CRediT authorship contribution statement

Qunfang Tang: Conceived the presented idea, Developed the theory and performed the computations, Drafted the article, Discussed the results and contributed to the final manuscript. Jie Yang: Conceived the presented idea, Critical revision of the article, Discussed the results and contributed to the final manuscript. Xiangjian He: Conceived the presented idea, Critical revision of the article, Discussed the results and contributed to the final manuscript. Wenjing Jia: Developed the theory and

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors would like to thank the authors of compared papers, who provided related images and original codes, and also the anonymous reviewers for their insightful comments and valuable suggestions. The authors gratefully acknowledge financial support from China Scholarship Council.

This work was supported by National Natural Science Foundation of China (No.51879211), Hunan Provincial Natural Science Foundation of China (No.2017JJ3053), Hunan Provincial Key Research and Development Project of

References (49)

  • Anon, .,...
  • Anon, .,...
  • Berman, D., Avidan, S., et al., 2016. Non-local image dehazing. In: Proceedings of the IEEE Conference on Computer...
  • CaiB. et al.

    Dehazenet: An end-to-end system for single image haze removal

    IEEE Trans. Image Process.

    (2016)
  • ChiangJ.Y. et al.

    Underwater image enhancement by wavelength compensation and dehazing (wcid)

    IEEE Trans. Image Process.

    (2012)
  • ChoiD.H. et al.

    Color image enhancement based on single-scale retinex with a jnd-based nonlinear filter

  • ChoiL.K. et al.

    Referenceless prediction of perceptual fog density and perceptual image defogging

    IEEE Trans. Image Process.

    (2015)
  • Deng, Z., Zhu, L., Hu, X., Fu, C.-W., Xu, X., Zhang, Q., Qin, J., Heng, P.-A., 2019. Deep multi-model fusion for...
  • FattalR.

    Dehazing using color-lines

    ACM Trans. Graph. (TOG)

    (2014)
  • FinlaysonG.D. et al.

    Shades of Gray and Colour Constancy

    (2004)
  • FuntB.V. et al.

    Retinex in matlab

    J. Electron. Imaging

    (2004)
  • Galdran, A., Alvarez-Gila, A., Bria, A., Vazquez-Corral, J., Bertalmo, M., 2018. On the duality between retinex and...
  • GaldranA. et al.

    Fusion-based variational image dehazing

    IEEE Signal Process. Lett.

    (2016)
  • HeK. et al.

    Single image haze removal using dark channel prior

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • Cited by (37)

    • Efficient image dehazing algorithm using multiple priors constraints

      2023, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      These algorithms consider the contrast reduction characteristics in hazy images and emphasise the enhancement in texture information in low-visibility images. Representative methods include histogram equalisation (HE) [3], retinex [4], wavelet transform [5], and homomorphic filtering [6]. However, these algorithms ignore the specific image-quality degradation mechanism, which limits the quality of the final output images.

    • Rapid nighttime haze removal with color-gray layer decomposition

      2022, Signal Processing
      Citation Excerpt :

      Notably, the reason why these approaches [20–22,24] can cause massive color artifacts is that color constraint in the glow-haze decomposition [20] fails for nighttime hazy scenes with severe color attenuation. Tang et al. [25] proposed an approach based on Retinex theory and Taylor series expansion to implement nighttime image dehazing. Recently, the learning-based methods [26–28] have made significant progress in nighttime haze removal.

    • Visibility enhancement and dehazing: Research contribution challenges and direction

      2022, Computer Science Review
      Citation Excerpt :

      RESIDE-SOTS, NYU-depth, Dense Haze, I-Haze, O-Haze, and SUN-Haze [219] are grouped as synthetic image dataset and RESIDE-RTTS, RESIDE-HSTS, Night-haze [220], some frequently used real-world image sets and our Dense to Low (D2L-Haze) real-world hazy image set are categorized into real-world image dataset. We choose 29 dense and 55 varicolored images from RESIDE-RTTS [158] dataset and 10 frequently used night and halation-effect images from SUN-HAZE [219] and night datasets [220]. . As dehazing approaches are sensitive to the sky and white regions, six real-world images are selected that are hard to handle and challenging for the evaluation of the dehazing approaches.

    View all citing articles on Scopus
    1

    Senior Member, IEEE.

    View full text