Skip to main content
SpringerLink
Log in

Learning to detect soft shadow from limited data

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Soft shadow is more challenging to detect than hard shadow due to its ambiguous boundary. Existing shadow detection methods pay more attention to hard shadow scene since collecting and annotating hard shadow images is more effortless. Motivated by that soft shadow has similar characteristics with hard shadow, and many traditional hard shadow datasets are publicly available, we propose a novel soft shadow detection method (namely Soft-DA) based on adversarial learning and domain adaptation scheme. Specifically, we create a limited soft shadow dataset, containing 1K soft shadow images with various scenes and shapes. Note that we just only need to annotate 0.4K shadow masks for semi-supervised learning. Besides, to tackle obvious domain discrepancy and potential intention difference between different datasets and similar tasks, we first align data distributions between domains by feature adversarial adaptation. And then, we introduce a novel detector separation strategy to tackle the intention difference issue. In this way, Soft-DA can effectively detect soft shadow with only a small number of soft shadow annotations. Extensive experiments demonstrate that our method can achieve superior performance to state of the arts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1202–1216 (2010)

    Article  Google Scholar 

  2. Bandara, R., Ranathunga, L., Abdullah, N.A.: Deep learned compact binary descriptor with a lightweight network-in-network architecture for visual description. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01798-5

    Article  Google Scholar 

  3. Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning, pp. 1180–1189 (2015)

  4. Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: BMVC, pp. 1–11 (2014)

  5. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  6. Gryka, M., Terry, M., Brostow, G.J.: Learning to remove soft shadows. ACM Trans. Graph. 34(5), 153 (2015)

    Article  Google Scholar 

  7. Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR 2011, pp. 2033–2040. IEEE (2011)

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  9. Hsu, H.K., Yao, C.H., Tsai, Y.H., Hung, W.C., Tseng, H.Y., Singh, M., Yang, M.H.: Progressive domain adaptation for object detection. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 749–757 (2020)

  10. Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7454–7462 (2018)

  11. Kán, P., Kafumann, H.: Deeplight: light source estimation for augmented reality using deep learning. The Vis. Comput. 35(6–8), 873–883 (2019)

    Article  Google Scholar 

  12. Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. (TOG) 30(6), 1–12 (2011)

    Article  Google Scholar 

  13. Khan, S.H., Bennamoun, M., Sohel, F., Togneri, R.: Automatic shadow detection and removal from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 431–446 (2015)

    Article  Google Scholar 

  14. Khodabandeh, M., Vahdat, A., Ranjbar, M., Macready, W.G.: A robust learning approach to domain adaptive object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 480–490 (2019)

  15. Lafarge, M.W., Pluim, J.P., Eppenhof, K.A., Moeskops, P., Veta, M.: Domain-adversarial neural networks to address the appearance variability of histopathology images. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 83–91. Springer (2017)

  16. Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating natural illumination from a single outdoor image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 183–190. IEEE (2009)

  17. Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+ D net: training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 680–696 (2018)

  18. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

  19. Long, M., Zhu, H., Wang, J., Jordan, M.I.: Deep transfer learning with joint adaptation networks. In: International Conference on Machine Learning, pp. 2208–2217 (2017)

  20. Luo, Y., Zheng, L., Guan, T., Yu, J., Yang, Y.: Taking a closer look at domain shift: category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507–2516 (2019)

  21. Luo, Z., Zou, Y., Hoffman, J., Fei-Fei, L.F.: Label efficient learning of transferable representations acrosss domains and tasks. In: Advances in Neural Information Processing Systems, pp. 165–177 (2017)

  22. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)

  23. Mohajerani, S., Saeedi, P.: Cpnet: a context preserver convolutional neural network for detecting shadows in single rgb images. In: 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–5 (2018)

  24. Motiian, S., Piccirilli, M., Adjeroh, D.A., Doretto, G.: Unified deep supervised domain adaptation and generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5715–5725 (2017)

  25. Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4510–4518 (2017)

  26. Nielsen, M., Madsen, C.B.: Graph cut based segmentation of soft shadows for seamless removal and augmentation. In: SCIA’07 Proceedings of the 15th Scandinavian Conference on Image Analysis, pp. 918–927 (2007)

  27. Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1693–1700. IEEE (2009)

  28. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)

  29. Saito, K., Watanabe, K., Ushiku, Y., Harada, T.: Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723–3732 (2018)

  30. Sanchez-Matilla, R., Poiesi, F., Cavallaro, A.: Online multi-target tracking with strong and weak detections. In: European Conference on Computer Vision, pp. 84–99. Springer (2016)

  31. Savarese, S., Rushmeier, H., Bernardini, F., Perona, P.: Shadow carving. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 1, pp. 190–197. IEEE (2001)

  32. Shen, L., Wee Chua, T., Leman, K.: Shadow optimization from structured deep edge detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2067–2074 (2015)

  33. Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167–7176 (2017)

  34. Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)

  35. Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: European Conference on Computer Vision, pp. 816–832. Springer (2016)

  36. Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1788–1797 (2018)

  37. Wang, S., Yu, L., Li, K., Yang, X., Fu, C.W., Heng, P.A.: Boundary and entropy-driven adversarial learning for fundus image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 102–110. Springer (2019)

  38. Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1880–1889 (2020)

  39. Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Natural shadow matting. ACM Trans. Graph. (TOG) 26(2), 8-es (2007)

    Article  Google Scholar 

  40. Zhang, Y., Chen, H., Wei, Y., Zhao, P., Cao, J., Fan, X., Lou, X., Liu, H., Hou, J., Han, X., et al.: From whole slide imaging to microscopy: deep microscopy adaptation network for histopathology cancer image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 360–368. Springer (2019)

  41. Zheng, M., Lei, Z., Zhang, K.: Intelligent detection of building cracks based on deep learning. Image Vis. Comput. 103, 103987 (2020)

    Article  Google Scholar 

  42. Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5167–5176 (2019)

  43. Zhu, L., Deng, Z., Hu, X., Fu, C.W., Xu, X., Qin, J., Heng, P.A.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 122–137 (2018)

  44. Zhu, X., Chen, Z.: Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network. Vis. Comput. 36, 1–13 (2019)

    Google Scholar 

Download references

Acknowledgements

This work is supported by the Natural Science Foundation of Xinjiang Autonomous Region in China (No. 2020D01A48) and the National Social Science Foundation Western Project (No. 20XGL029).

Author information

Author notes

  1. Wen Wu and Shuping Zhang contributed equally to this work.

Authors and Affiliations

  1. School of Information Engineering, Xinjiang Institute of Technology, Aksu, 843100, China

    Wen Wu, Shuping Zhang & Mi Tian

  2. School of Computer and Information Engineering, Hubei University, Wuhan, 431400, China

    Daoqiang Tan

  3. College of Information Science and Engineering, Xinjiang University, Ürümqi, 830046, China

    Xiantao Wu

  4. School of Electrical and Electronic Engineering, Wenzhou University, Wenzhou, 325035, China

    Yi Wan

Authors
  1. Wen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mi Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daoqiang Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiantao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yi Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Wan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, W., Zhang, S., Tian, M. et al. Learning to detect soft shadow from limited data. Vis Comput 38, 1665–1675 (2022). https://doi.org/10.1007/s00371-021-02095-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02095-5

Keywords

Search

Navigation