Abstract
The quality of photos is highly susceptible to severe weather such as heavy rain; it can also degrade the performance of various visual tasks like object detection. Rain removal is a challenging problem because rain streaks have different appearances even in one image. Regions where rain accumulates appear foggy or misty, while rain streaks can be clearly seen in areas where rain is less heavy. We propose removing various rain effects in pictures using a hybrid multiscale loss guided multiple feature fusion de-raining network (MSGMFFNet). Specially, to deal with rain streaks, our method generates a rain streak attention map, while preprocessing uses gamma correction and contrast enhancement to enhanced images to address the problem of rain accumulation. Using these tools, the model can restore a result with abundant details. Furthermore, a hybrid multiscale loss combining L1 loss and edge loss is used to guide the training process to pay attention to edge and content information. Comprehensive experiments conducted on both synthetic and real-world datasets demonstrate the effectiveness of our method.
Article PDF
Similar content being viewed by others
References
Zhang, H.; Patel, V. M. Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 695–704, 2018.
Ren, D. W.; Zuo, W. M.; Hu, Q. H.; Zhu, P. F.; Meng, D. Y. Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3937–3946, 2019.
Chen, D. Y.; Chen, C. C.; Kang, L. W. Visual depth guided color image rain streaks removal using sparse coding. IEEE Transactions on Circuits and Systems for Video Technology Vol. 24, No. 8, 1430–1455, 20
Chen, Y. L.; Hsu, C. T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE International Conference on Computer Vision, 1968–1975, 2013.
Fu, X. Y.; Huang, J. B.; Zeng, D. L.; Huang, Y.; Ding, X. H.; Paisley, J. Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1715–1723, 2017.
Fu, X. Y.; Huang, J. B.; Ding, X. H.; Liao, Y. H.; Paisley, J. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing Vol. 26, No. 6, 2944–2956, 20
Kang, L. W.; Lin, C. W.; Fu, Y. H. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing Vol. 21, No. 4, 1742–1755, 20
Li, X.; Wu, J. L.; Lin, Z. C.; Liu, H.; Zha, H. B. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Computer Vision — ECCV 2018. Type="Italic">Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds}. Springer Cham, 262–277
Li, Y.; Tan, R. T.; Guo, X. J.; Lu, J. B.; Brown, M. S. Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2736–2744, 2016.
Yasarla, R.; Patel, V. M. Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8405–8414, 2019.
Wang, T.; Yang, X.; Xu, K.; Chen, S. Z.; Zhang, Q.; Lau, R. W. H. Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12270–12279, 2019.
Yang, W. H.; Tan, R. T.; Feng, J. S.; Liu, J. Y.; Guo, Z. M.; Yan, S. C. Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1685–1694, 2017.
Luo, Y.; Xu, Y.; Ji, H. Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, 3397–3405, 2015.
Zhang, H.; Sindagi, V.; Patel, V. M. Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology Vol. 30, No. 11, 3943–3956, 20
Tan, J.; Zhang, Y.; Fu, H. Y.; Ma, H. D.; Gao, N. Multi-scale attentive residual network for single image deraining. In: Human Centered Computing. Lecture Notes in Computer Science, Vol. 11956. Milošević, D.; Tang, Y.; Zu, Q. Eds. Springer Cham, 351–362
Eigen, D.; Krishnan, D.; Fergus, R. Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE International Conference on Computer Vision, 633–640, 2013.
Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Computation Vol. 9, No. 8, 1735–1780, 19
Zhang, K. H.; Luo, W. H.; Ren, W. Q.; Wang, J. W.; Zhao, F.; Ma, L.; Li, H. Beyond monocular deraining: Stereo image deraining via semantic understanding. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12372. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 71–89
Ren, W. Q.; Ma, L.; Zhang, J. W.; Pan, J. S.; Cao, X. C.; Liu, W.; Yang, M. Gated fusion network for single image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3253–3261, 2018.
Ren, W. Q.; Pan, J. S.; Zhang, H.; Cao, X. C.; Yang, M. H. Single image dehazing via multi-scale convolutional neural networks with holistic edges. International Journal of Computer Vision Vol. 128, No. 1, 240–259, 20
Wang, F.; Jiang, M. Q.; Qian, C.; Yang, S.; Li, C.; Zhang, H. G.; Wang, X.; Tang, X. Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6450–6458, 2017.
Kim, J.; Lee, J. K.; Lee, K. M. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1637–1645, 2016.
Tai, Y.; Yang, J.; Liu, X. M. Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2790–2798, 2017.
Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 1724–1734, 2014.
Ancuti, C. O.; Ancuti, C. Single image dehazing by multi-scale fusion. IEEE Transactions on Image Processing Vol. 22, No. 8, 3271–3282, 20
Choi, L. K.; You, J.; Bovik, A. C. Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Transactions on Image Processing Vol. 24, No. 11, 3888–3901, 20
Chen, D. D.; He, M. M.; Fan, Q. N.; Liao, J.; Zhang, L. H.; Hou, D. D.; Yuan, L.; Hua, G. Gated context aggregation network for image dehazing and deraining. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1375–1383, 2019.
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778, 2016.
Huang, D. A.; Kang, L. W.; Yang, M. C.; Lin, C. W.; Wang, Y. C. F. Context-aware single image rain removal. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 164–169, 2012.
Johnson, J.; Alahi, A.; Li, F. F. Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision — ECCV 2016. Lecture Notes in Computer Science, Vol. 9906. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 694–711
Zhao, H.; Gallo, O.; Frosio, I.; Kautz, J. Loss functions for neural networks for image processing. arXiv preprint] arXiv:1511.08861, 2015
HuynhThu, Q.; Ghanbari, M. Scope of validity of PSNR in image/video quality assessment. Electronics Letters Vol. 44, No. 13, 800–801, 20
Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600–612, 20
Li, S.; Araujo, I. B.; Ren, W.; Wang, Z.; Tokuda, E. K.; Junior, R. H.; Cesar-Junior, R.; Zhang, J.; Guo, X.; Cao, X. Single image deraining: A comprehensive benchmark analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3838–3847, 2019.
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint] arXiv:1412.6980, 2014
Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 6, 1137–1149, 20
Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.; Berg, A. C. SSD: Single shot multibox detector. In: Proceedings of the 14th European Conference on Computer Vision, 21–37, 2016.
Lin, T.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, 2980–2988, 2017.
Chen, K.; Wang, J.; Pang, J.; Cao, Y.; Xiong, Y.; Li, X.-X.; Sun, S.-Y.; Feng, W.; Liu, Z.-W.; Xu, J. et al. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint] arXiv:1906.07155, 2019
Acknowledgements
This work was supported in part by the National Key R&D Program of China under No. 2017YFB1003000, the National Natural Science Foundation of China under No. 61872047 and No. 61720106007, the Beijing Nova Program under No. Z201100006820124, the Beijing Natural Science Foundation (L191004), and the 111 Project (B18008).
Author information
Authors and Affiliations
Corresponding author
Additional information
Huiyuan Fu received his Ph.D. degree in computer science from Beijing University of Posts and Telecommunications, China, in 2014. He is an associate professor at the School of Computer Science, Beijing University of Posts and Telecommunications. His research area includes visual big data, machine learning and pattern recognition, multimedia systems, etc. He received a Best Student Paper Award at ICME in 2016.
Yu Zhang is a master student in the Department of Computer Science, Beijing University of Posts and Telecommunications. Her research interests include machine learning and pattern recognition.
Huadong Ma received his Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences (CAS), in 1995, his M.S. degree in computer science from Shenyang Institute of Computing Technology, CAS, in 1990, and his B.S. degree in mathematics from Henan Normal University, China, in 1984. He is a professor at the School of Computer Science, Beijing University of Posts and Telecommunications. His research interests include multimedia networks and systems, the Internet of things, and sensor networks. He has published over 300 papers in these fields.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Fu, H., Zhang, Y. & Ma, H. See clearly on rainy days: Hybrid multiscale loss guided multi-feature fusion network for single image rain removal. Comp. Visual Media 7, 467–482 (2021). https://doi.org/10.1007/s41095-021-0210-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0210-3