Skip to main content
SpringerLink
Log in

Single image deraining via deep pyramid network with spatial contextual information aggregation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Rain streaks usually give rise to visual degradation and cause many computer vision algorithms to fail. So it is necessary to develop an effective deraining algorithm as preprocess of high-level vision tasks. In this paper, we propose a novel deep learning based deraining method. Specifically, the multi-scale kernels and feature maps are both important for single image deraining. However, the previous works ignore the two multi-scale information or only consider the multi-scale kernels information. Instead, our method learns multi-scale information both from the perspectives of kernels and feature maps, respectively, by designing spatial contextual information aggregation module and pyramid network module. The former module can capture the rain streaks with different sizes and the latter module can extract rain streaks from different scales further. Moreover, we also employ squeeze-and-excitation and skip connections to enhance the correlation between channels and transmit the information from low-level to high-level, respectively. The experimental results show that the proposed method achieves significant improvements over the recent state-of-the-art methods in Rain100H, Rain100L, Rain1200 and Rain1400 datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Brewer N, Liu N (2008) Using the shape characteristics of rain to identify and remove rain from video. In: Structural, syntactic, and statistical pattern recognition, pp 451–458. https://doi.org/10.1007/978-3-540-89689-0_49

  2. Chen J, Tan C, Hou J, Chau L, Li H (2018) Robust video content alignment and compensation for rain removal in a CNN framework. In: CVPR, pp 6286–6295. https://doi.org/10.1109/CVPR.2018.00658. http://openaccess.thecvf.com/content_cvpr_2018/html/Chen_Robust_Video_content_cvpr_2018_paper.html

  3. Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV, pp 1968–1975. https://doi.org/10.1109/ICCV.2013.247

  4. Chen Y, Wang Z, Peng Y, Zhang Z, Yu G, Sun J (2018) Cascaded pyramid network for multi-person pose estimation. In: CVPR, pp 7103–7112. https://doi.org/10.1109/CVPR.2018.00742. http://openaccess.thecvf.com/content_cvpr_2018/html/Chen_Cascaded_Pyramid_Network_CVPR_2018_paper.html

  5. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal 26(6), 2944–2956. https://doi.org/10.1109/TIP.2017.2691802

  6. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: CVPR, pp 1715–1723. https://doi.org/10.1109/CVPR.2017.186

  7. Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: CVPR, pp 528–535. https://doi.org/10.1109/CVPR.2004.79

  8. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: CVPR, pp 7132–7141. https://doi.org/10.1109/CVPR.2018.00745

  9. Huang D, Kang L, Yang M, Lin C, Wang Y (2012) Context-aware single image rain removal. In: ICME, pp 164–169. https://doi.org/10.1109/ICME.2012.92

  10. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  11. Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition 21(4), 1742–1755. https://doi.org/10.1109/TIP.2011.2179057

  12. Kim J, Lee C, Sim J, Kim C (2013) Single-image deraining using an adaptive nonlocal means filter. In: ICIP, pp 914–917. https://doi.org/10.1109/ICIP.2013.6738189

  13. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. In: CoRR, vol. arXiv:1412.6980

  14. Li G, He X, Zhang W, Chang H, Dong L, Lin L (2018) Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM MM, pp 1056–1064. https://doi.org/10.1145/3240508.3240636

  15. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp 262–277. https://doi.org/10.1007/978-3-030-01234-2_16

  16. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: CVPR, pp 2736–2744. https://doi.org/10.1109/CVPR.2016.299

  17. Lin T, Dollár P, Girshick RB, He K, Hariharan B, Belongie SJ (2017) Feature pyramid networks for object detection. In: CVPR, pp 936–944. https://doi.org/10.1109/CVPR.2017.106

  18. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: ICCV, pp. 3397–3405. https://doi.org/10.1109/ICCV.2015.388

  19. Ranjan A, Black MJ (2017) Optical flow estimation using a spatial pyramid network. In: CVPR, pp 2720–2729. https://doi.org/10.1109/CVPR.2017.291

  20. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(4):640–651. https://doi.org/10.1109/TPAMI.2016.2572683

    Article  Google Scholar 

  21. Tripathi AK, Mukhopadhyay S (2014) Removal of rain from videos: a review. SIViP 8(8):1421–1430. https://doi.org/10.1007/s11760-012-0373-6

    Article  Google Scholar 

  22. Wang C, Wang H, Su Z, Yang Y (2019) Embedding non-local mean in squeeze-and-excitation network for single image deraining. In: ICMEW, pp 264–269. https://doi.org/10.1109/ICMEW.2019.00-76

  23. Wang C, Zhang M, Pan J, Su Z (2019) Single image rain removal via densely connected contextual and semantic correlation net. J Electron Imag 28(3):033018. https://doi.org/10.1117/1.JEI.28.3.033018

    Google Scholar 

  24. Wang C, Zhang M, Su Z, Wu Y, Yao G, Wang H (2019) Learning a multi-level guided residual network for single image deraining. Signal Process Imag Commun 78:206–215. https://doi.org/10.1016/j.image.2019.07.003. http://www.sciencedirect.com/science/article/pii/S0923596519305582

    Article  Google Scholar 

  25. Wang C, Zhang M, Su Z, Yao G, Wang Y, Sun X, Luo X (2019) From coarse to fine: a stage-wise deraining net. IEEE Access 7:84420–84428. https://doi.org/10.1109/ACCESS.2019.2922549

    Article  Google Scholar 

  26. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Processing 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  27. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: CVPR, pp 1685–1694. https://doi.org/10.1109/CVPR.2017.183

  28. Zhang H, Patel VM (2018) Densely connected pyramid dehazing network. In: CVPR, pp 3194–3203. https://doi.org/10.1109/CVPR.2018.00337. http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Densely_Connected_Pyramid_CVPR_2018_paper.html

  29. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: CVPR, pp 695–704. https://doi.org/10.1109/CVPR.2018.00079. http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Density-Aware_Single_Image_CVPR_2018_paper.html

  30. Zhang H, Sindagi V, Patel VM (2017) Image de-raining using a conditional generative adversarial network. In: CoRR, vol arXiv:1701.05957

  31. Zhang X, Li H, Qi Y, Leow WK, Ng TK (2006) Rain removal in video by combining temporal and chromatic properties. In: ICME, pp 461–464. https://doi.org/10.1109/ICME.2006.262572

Download references

Author information

Authors and Affiliations

  1. Dalian University of Technology, Dalian, China

    Cong Wang, Yutong Wu, Yu Cai, Zhixun Su & Hongyan Wang

  2. Chengdu University of Technology, Chengdu, China

    Guangle Yao

  3. Guilin University of Electronic Technology, Guilin, China

    Zhixun Su

  4. School of Mathematical Sciences, Dalian University of Technology, Dalian, China

    Zhixun Su

Authors
  1. Cong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yutong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangle Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhixun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongyan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhixun Su.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the Natural Science Foundation of China [grant numbers 61572099]; Major National Science and Technology Project of China [grant number 2018ZX04016001-011].

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Wu, Y., Cai, Y. et al. Single image deraining via deep pyramid network with spatial contextual information aggregation. Appl Intell 50, 1437–1447 (2020). https://doi.org/10.1007/s10489-019-01567-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-019-01567-5

Keywords

Search

Navigation