Skip to main content
SpringerLink
Log in

Successive Graph Convolutional Network for Image De-raining

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Deep convolutional neural networks (CNNs) have shown their advantages in the single image de-raining task. However, most existing CNNs-based methods utilize only local spatial information without considering long-range contextual information. In this paper, we propose a graph convolutional networks (GCNs)-based model to solve the above problem. We specifically design two graphs to extract representations from new dimensions. The first graph models the global spatial relationship between pixels in the feature, while the second graph models the interrelationship across the channels. By integrating conventional CNNs and our GCNs into a single framework, the proposed method is able to explore comprehensive feature representations from three aspects, i.e., local spatial patterns, global spatial coherence and channel correlation. To better exploit the explored rich feature representations, we further introduce a simple yet effective recurrent operations to perform the de-raining process in a successive manner. Benefiting from the rich information exploration and exploitation, our method achieves state-of-the-art results on both synthetic and real-world data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Notes

  1. https://www.photoshopessentials.com/photo-effects/photoshop-weather-effects-rain/

  2. https://www.clarifai.com/

References

  • Abadi, M., Agarwal, A., & Barham, P., et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 1603.04467 (2016)

  • Arbelaez, P., Maire, M., Fowlkes, C., & Malik, J. (2010). Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 898–916.

    Article  Google Scholar 

  • Barnum, P. C., Narasimhan, S., & Kanade, T. (2010). Analysis of rain and snow in frequency space. International Journal of Computer Vision, 86(2), 256–274.

    Article  Google Scholar 

  • Bossu, J., Hautière, N., & Tarel, J. P. (2011). Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International Journal of Computer Vision, 93(3), 348–367.

    Article  Google Scholar 

  • Chang, Y., Yan, L., & Zhong, S. (2017). Transformed low-rank model for line pattern noise removal. In: ICCV

  • Chen, C., Xiong, Z., Tian, X., Zha, Z. J., & Wu, F. (2019). Real-world image denoising with deep boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence,. https://doi.org/10.1109/TPAMI.2019.2921548.

    Article  Google Scholar 

  • Chen, J., & Chau, L. P. (2013). A rain pixel recovery algorithm for videos with highly dynamic scenes. IEEE Transactions on Image Processing, 23(3), 1097–1104.

    Article  MathSciNet  MATH  Google Scholar 

  • Chen, J., Tan, C.H., Hou, J., Chau, L.P., & Li, H. (2018). Robust video content alignment and compensation for rain removal in a cnn framework. In: CVPR

  • Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.

    Article  Google Scholar 

  • Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: A\(\hat{2}\)-nets: Double attention networks. In: NeurIPS (2018)

  • Chen, Y., & Pock, T. (2016). Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1256–1272.

    Article  Google Scholar 

  • Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., & Kalantidis, Y. (2019). Graph-based global reasoning networks. In: CVPR

  • Chen, Y.L., & Hsu, C.T. (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV

  • Dong, C., Deng, Y., Change Loy, C., & Tang, X. (2015). Compression artifacts reduction by a deep convolutional network. In: ICCV, pp. 576–584

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.

    Article  Google Scholar 

  • Eigen, D., Krishnan, D., & Fergus, R. (2013). Restoring an image taken through a window covered with dirt or rain. In: ICCV

  • Fan, Q., Chen, D., Yuan, L., Hua, G., Yu, N., & Chen, B. (2018). Decouple learning for parameterized image operators. In: ECCV, pp. 442–458

  • Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017). Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944–2956.

    Article  MathSciNet  MATH  Google Scholar 

  • Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017). Removing rain from single images via a deep detail network. In: CVPR

  • Fu, X., Liang, B., Huang, Y., Ding, X., & Paisley, J. (2019). Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems,. https://doi.org/10.1109/TNNLS.2019.2926481.

    Article  Google Scholar 

  • Garg, K., & Nayar, S.K. (2004). Detection and removal of rain from videos. In: CVPR

  • Garg, K., & Nayar, S.K. (2005) When does a camera see rain? In: ICCV

  • Garg, K., & Nayar, S. K. (2007). Vision and rain. International Journal of Computer Vision, 75(1), 3–27.

    Article  Google Scholar 

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) Generative adversarial nets. In: NeurIPS, pp. 2672–2680

  • Gu, S., Meng, D., Zuo, W., & Zhang, L. (2017). Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: ICCV

  • He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6), 1397–1409.

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016) Deep residual learning for image recognition. In: CVPR

  • Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In: CVPR

  • Hu, X., Fu, C.W., Zhu, L., & Heng, P.A. (2019). Depth-attentional features for single-image rain removal. In: CVPR

  • Huang, D. A., Kang, L. W., Wang, Y. C. F., & Lin, C. W. (2014). Self-learning based image decomposition with applications to single image denoising. IEEE Transactions on Multimedia, 16(1), 83–93.

    Article  Google Scholar 

  • Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., & Jiang, J. (2020). Multi-scale progressive fusion network for single image deraining. In: CVPR

  • Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., & Wang, Y. (2017). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In: CVPR

  • Jiang, T. X., Huang, T. Z., Zhao, X. L., Deng, L. J., & Wang, Y. (2018). Fastderain: A novel video rain streak removal method using directional gradient priors. IEEE Transactions on Image Processing, 28(4), 2089–2102.

    Article  MathSciNet  Google Scholar 

  • Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp. 694–711

  • Johnson, J., Gupta, A., & Fei-Fei, L. (2018). Image generation from scene graphs. In: CVPR

  • Kang, L. W., Lin, C. W., & Fu, Y. H. (2012). Automatic single image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4), 1742–1755.

    Article  MathSciNet  MATH  Google Scholar 

  • Kim, J.H., Lee, C., Sim, J.Y., & Kim, C.S. (2013). Single-image deraining using an adaptive nonlocal means filter. In: IEEE ICIP

  • Kim, J. H., Sim, J. Y., & Kim, C. S. (2015). Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, 24(9), 2658–2670.

    Article  MathSciNet  MATH  Google Scholar 

  • Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. In: ICLR

  • Kipf, T.N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In: ICLR

  • Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: NeurIPS

  • Li, G., He, X., Zhang, W., Chang, H., Dong, L., & Lin, L. (2018). Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM MM

  • Li, J., Ma, A. J., & Yuen, P. C. (2018). Semi-supervised region metric learning for person re-identification. International Journal of Computer Vision, 126(8), 855–874.

    Article  Google Scholar 

  • Li, L., Pan, J., Lai, W. S., Gao, C., Sang, N., & Yang, M. H. (2019). Blind image deblurring via deep discriminative priors. International Journal of Computer Vision, 127(8), 1025–1043.

    Article  Google Scholar 

  • Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., & Meng, D. (2018). Video rain streak removal by multiscale convolutional sparse coding. In: CVPR

  • Li, R., Cheong, L.F., & Tan, R.T. (2019). Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: CVPR

  • Li, R., Tan, R.T., Cheong, L.F., Aviles-Rivero, A.I., Fan, Q., & Schonlieb, C.B. (2019). Rainflow: Optical flow under rain streaks and rain veiling effect. In: ICCV

  • Li, S., Araujo, I.B., Ren, W., Wang, Z., Tokuda, E.K., Junior, R.H., Cesar-Junior, R., Zhang, J., Guo, X., & Cao, X. (2019). Single image deraining: A comprehensive benchmark analysis. In: CVPR

  • Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV

  • Li, Y., Ouyang, W., Zhou, B., Shi, J., Zhang, C., & Wang, X. (2018). Factorizable net: an efficient subgraph-based framework for scene graph generation. In: ECCV

  • Li, Y., Tan, R.T., Guo, X., Lu, J., & Brown, M.S. (2016). Rain streak removal using layer priors. In: CVPR

  • Lin, M., Chen, Q., & Yan, S. (2014). Network in network. In: ICLR

  • Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). D3R-Net: Dynamic routing residue recurrent network for video rain removal. IEEE Transactions on Image Processing, 28(2), 699–712.

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In: CVPR

  • Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., et al. (2019). Deep learning for generic object detection: A survey. International Journal of Computer Vision,. https://doi.org/10.1007/s11263-019-01247-4.

    Article  Google Scholar 

  • Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In: ICCV

  • Mordan, T., Thome, N., Henaff, G., & Cord, M. (2019). End-to-end learning of latent deformable part-based representations for object detection. International Journal of Computer Vision, 127(11–12), 1659–1679.

    Article  Google Scholar 

  • Narasimhan, S. G., & Nayar, S. K. (2002). Vision and the atmosphere. International Journal of Computer Vision, 48(3), 233–254.

    Article  MATH  Google Scholar 

  • Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017). 3d graph neural networks for rgbd semantic segmentation. In: ICCV

  • Qian, R., Tan, R.T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In: CVPR

  • Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In: CVPR

  • Ren, W., Pan, J., Zhang, H., Cao, X., & Yang, M. H. (2019). Single image dehazing via multi-scale convolutional neural networks with holistic edges. International Journal of Computer Vision,. https://doi.org/10.1007/s11263-019-01235-8.

    Article  Google Scholar 

  • Ren, W., Tian, J., Han, Z., Chan, A., & Tang, Y. (2017). Video desnowing and deraining based on matrix decomposition. In: ICCV

  • Romano, Y., & Elad, M. (2015). Boosting of image denoising algorithms. SIAM Journal on Imaging Sciences, 8(2), 1187–1219.

    Article  MathSciNet  MATH  Google Scholar 

  • Sakaridis, C., Dai, D., & Van Gool, L. (2018). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9), 973–992.

    Article  Google Scholar 

  • Santhaseelan, V., & Asari, V. K. (2015). Utilizing local phase information to remove rain from video. International Journal of Computer Vision, 112(1), 71–89.

    Article  Google Scholar 

  • Shao, Y., Li, L., Ren, W., Gao, C., & Sang, N. (2020). Domain adaptation for image dehazing. In: CVPR

  • Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A.A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In: CVPR

  • Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: CVPR

  • Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In: CVPR

  • Wang, C., Li, Z., Wu, J., Fan, H., Xiao, G., & Zhang, H. (2020). Deep residual haze network for image dehazing and deraining. IEEE Access, 8, 9488–9500.

    Article  Google Scholar 

  • Wang, G., Sun, C., & Sowmya, A. (2019). Erl-net: Entangled representation learning for single image de-raining. In: ICCV

  • Wang, H., Li, M., Wu, Y., Zhao, Q., & Meng, D. (2020). A survey on rain removal from video and single image. International Journal of Machine Learning and Cybernetics,. https://doi.org/10.1007/s13042-020-01061-2.

    Article  Google Scholar 

  • Wang, H., Xie, Q., Zhao, Q., & Meng, D. (2020). A model-driven deep neural network for single image rain removal. In: CVPR

  • Wang, H., Zhu, X., Gong, S., & Xiang, T. (2018). Person re-identification in identity regression space. International Journal of Computer Vision, 126(12), 1288–1310.

    Article  Google Scholar 

  • Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R.W. (2019). Spatial attentive single-image deraining with a high quality real rain dataset. In: CVPR

  • Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In: CVPR

  • Wang, Y., Liu, S., Chen, C., & Zeng, B. (2017). A hierarchical approach for rain or snow removing in a single color image. IEEE Transactions on Image Processing, 26(8), 3936–3950.

    Article  MathSciNet  MATH  Google Scholar 

  • Wei, W., Meng, D., Zhao, Q., Wu, C., & Xu, Z. (2019). Semi-supervised transfer learning for image rain removal. In: CVPR

  • Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., & Xu, Z. (2017). Should we encode rain streaks in video as deterministic or stochastic? In: ICCV

  • Wojna, Z., Ferrari, V., Guadarrama, S., Silberman, N., Chen, L. C., Fathi, A., et al. (2019). The devil is in the decoder: Classification, regression and gans. International Journal of Computer Vision, 127(11–12), 1694–1706.

    Article  Google Scholar 

  • Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., & Woo, W.c. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In: NeurIPS

  • Yang, W., Liu, J., Yang, S., & Guo, Z. (2019). Scale-free single image deraining via visibility-enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6), 2948–2961.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017). Deep joint rain detection and removal from a single image. In: CVPR

  • Yang, W., Tan, R. T., Feng, J., Liu, J., Yan, S., & Guo, Z. (2019). Joint rain detection and removal from a single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6), 1377–1393.

    Article  Google Scholar 

  • Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2020). Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence,. https://doi.org/10.1109/TPAMI.2020.2995190.

    Article  Google Scholar 

  • Yasarla, R., & Patel, V.M. (2019). Uncertainty guided multi-scale residual learning using a cycle spinning cnn for single image de-raining. In: CVPR

  • Yasarla, R., & Patel, V. M. (2020). Confidence measure guided single image de-raining. IEEE Transactions on Image Processing, 29, 4544–4555.

    Article  Google Scholar 

  • Yim, C., & Bovik, A. C. (2010). Quality assessment of deblocked images. IEEE Transactions on Image Processing, 20(1), 88–98.

    MathSciNet  MATH  Google Scholar 

  • Yu, F., & Koltun, V. (2016). Multi-scale context aggregation by dilated convolutions. In: ICLR

  • Zhang, H., & Patel, V. M. (2016). Sparse representation-based open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8), 1690–1696.

    Article  Google Scholar 

  • Zhang, H., & Patel, V.M. (2017). Convolutional sparse and low-rank coding-based rain streak removal. In: IEEE Winter Conference on Applications of Computer Vision

  • Zhang, H., & Patel, V.M. (2018). Densely connected pyramid dehazing network. In: CVPR

  • Zhang, H., & Patel, V.M. (2018). Density-aware single image de-raining using a multi-stream dense network. In: CVPR

  • Zhang, H., Riggan, B. S., Hu, S., Short, N. J., & Patel, V. M. (2019). Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks. International Journal of Computer Vision, 127(6–7), 845–862.

    Article  Google Scholar 

  • Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology,. https://doi.org/10.1109/TCSVT.2019.2920407.

    Article  Google Scholar 

  • Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7), 3142–3155.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, T., Ghanem, B., Liu, S., & Ahuja, N. (2013). Robust visual tracking via structured multi-task sparse learning. International Journal of Computer Vision, 101(2), 367–383.

    Article  MathSciNet  Google Scholar 

  • Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018). Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: CVPR

  • Zheng, X., Liao, Y., Guo, W., Fu, X., & Ding, X. (2013). Single-image-based rain and snow removal using multi-guided filter. In: International Conference on Neural Information Processing

  • Zhu, L., Fu, C.W., Lischinski, D., & Heng, P.A. (2017). Joint bi-layer optimization for single-image rain streak removal. In: ICCV

Download references

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant 2020AAA0105702, the National Natural Science Foundation of China (NSFC) under Grants U19B2038, 61620106009 and 61901433, the USTC Research Funds of the Double First-Class Initiative under Grant YD2100002003.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, University of Science and Technology of China, Hefei, China

    Xueyang Fu, Zheng-Jun Zha & Feng Wu

  2. School of Informatics, Xiamen University, Xiamen, China

    Qi Qi & Xinghao Ding

  3. Department of Electrical Engineering and Data Science Institute, Columbia University, New York, USA

    John Paisley

Authors
  1. Xueyang Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zheng-Jun Zha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinghao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Feng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. John Paisley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheng-Jun Zha.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Vishal Patel.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, X., Qi, Q., Zha, ZJ. et al. Successive Graph Convolutional Network for Image De-raining. Int J Comput Vis 129, 1691–1711 (2021). https://doi.org/10.1007/s11263-020-01428-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01428-6

Keywords

Search

Navigation