SDRNet: An end-to-end shadow detection and removal network

doi:10.1016/j.image.2020.115832

Signal Processing: Image Communication

Volume 84, May 2020, 115832

https://doi.org/10.1016/j.image.2020.115832 Get rights and content

Highlights

•
SDRNet can complete the shadow detection and removal tasks in a unified network.
•
SDRNet highly shares features and have no additional computational complexity.
•
A multi-scale global feature module is used to solve shadow area problems.
•
Shadow segmentation accuracy can be improved by using the image direction feature.
•
Superior performance can be achieved in both shadow detection and removal tasks.

Abstract

Image shadow detection and removal can effectively recover image information lost in the image due to the existence of shadows, which helps improve the accuracy of object detection, segmentation and tracking. Thus, aiming at the problem of the scale of the shadow in the image, and the inconsistency of the shadowed area with the original non-shadowed area after the shadow is removed, the multi-scale and global feature (MSGF) is used in the proposed method, combined with the non-local network and dense dilated convolution pyramid pooling network. Besides, aiming at the problem of inaccurate detection of weak shadows and complicated shape shadows in existing methods, the direction feature (DF) module is adopted to enhance the features of the shadow areas, thereby improving shadow segmentation accuracy. Based on the above two methods, an end-to-end shadow detection and removal network SDRNet is proposed. SDRNet completes the task of sharing two feature heights in a unified network without adding additional calculations. Experimental results on the two public datasets ISDT and SBU demonstrate that the proposed method achieves more than 10% improvement in the BER index for shadow detection and the RMSE index for shadow removal, which proves that the proposed SDRNet based on the MSGF module and DF module can achieve the best results compared with other existing methods.

Introduction

Image acquisition is the prerequisite for a variety of tasks. However, when acquiring images, external factors such as illumination and object occlusion results in shadows in the image. The existence of image shadows has advantages as well as disadvantages. The shadow can preserve the dynamic condition of the light scene and the valuable information of the objects in it. However, the shadow is a degrading phenomenon relative to the image, which can result in image information loss and influence the true information restoration of the original scene. It not only intuitively influences our observation for the image but also leads to subsequent image processing and analysis errors. Shadows are caused by an object occluding the light source, which is often combined with the corresponding occlusion object, has a very similar contour and is detrimental to the computer vision task. For instance, in the medical image processing field, shadows can interfere with lesion area segmentation and influence diagnosis. In the field of aerial image processing, shadows can influence ground target extraction, target matching, ground target tracking and many other tasks [1], [2]. For automatic driving, the existence of shadows can seriously influence the detection and segmentation of landmarks, stop lines, and lane lines [3]. To address the influence of image shadow and improve the practicality for a variety of applications, it is necessary to detect shadows in images, reconstruct the shadow area and remove the shadow, thereby reducing or eliminating the impact of the shadow on the computer vision task.

Shadow detection and removal consists of detection and removal of shadows in moving images (video sequences) and single images. The moving image can detect the shadow area by combining the relationship between the before-and-after-frame and then separating the shadow area from the object so that the target can be extracted accurately and conveniently. This method is mainly used for object tracking, such as vehicle tracking and human body tracking. Shadow removal for a single image is more complicated and more difficult because there is no before-and-after-frame information available. Thus, we mainly focus on this situation. Although there are many relevant studies worldwide, existing algorithms generally do not have universality; that is, they are only suitable for some specific scenarios. Therefore, studying a general single image shadow detection and removal algorithm that is suitable for a variety of scenes is still a challenging task. The contributions of this paper are mainly as follows.

(1) Considering the problem of the demand of the change in the shadow area for different receptive fields during the shadow detection process, and the inconsistency of the shadow area style and original non-shadowed area style in the de-shadowed images obtained by the existing de-shadowing algorithms, inspired by the non-local network and dense dilated convolutional pyramid pooling network, we propose a new module named the multi-scale and global feature (MSGF).

(2) Considering the problem of weak shadows and the complicated shape of shadows are difficult to accurately segment during the shadow detection process, we propose using the DF module to extract the image direction feature. By using the DF, the shadow area features can be enhanced, and the non-shadow area feature can be weakened. In addition, the interface between the shadow area and the non-shadow area can be strengthened, and the accuracy of the shadow segmentation can be greatly improved.

(3) We propose an end-to-end shadow detection and shadow removal network named SDRNet. SDRNet completes two tasks in a unified network, which highly share features and have no additional computational complexity. Experiments on public databases prove that SDRNet exceeds the existing methods in both shadow detection and shadow removal tasks, and optimum performance can be achieved by using SDRNet.

Section snippets

Related works

Most image de-shadowing studies first detect the shadow area in the image and then restore the shadow area. This kind of method can be divided into physical-based algorithms and deep learning-based algorithms.

SDRNet network structure

An end-to-end shadow detection and removal network SDRNet is proposed in this paper, which is different from the one-stage method and the two-stage method mentioned above. SDRNet integrates shadow detection and shadow removal into one network, which makes it possible for shadow detection and shadow removal to share features, greatly reducing the number of calculations and improving the efficiency of the model. At the same time, the joint training of the shadow detection network and shadow

Experimental results and analysis

In this section, we mainly conduct experiments on SDRNet and compare the shadow detection and shadow removal results with those of the existing optimal methods and separately perform quantitative and qualitative analyses.

Discussion

Image shadow detection and removal face many situations, apart from inaccurate shadow detection of weak shadows, complicated shadow shapes, and style inconsistency between shadow and non-shadow areas, overlapping shadows with different intensities is also a common phenomenon. In our experiment, we find that overlapping shadows with different intensities is in fact still a shadow. Possible situations including many weak shadows overlaps with each other may become a strong shadow. While the weak

Conclusion

As an important part of image pre-processing, image shadow detection and removal can effectively recover lost image information caused by the existence of shadows, which is helpful for improving the accuracy of object detection, segmentation and tracking. Therefore, we proposed an end-to-end shadow detection and removal network SDRNet to deal with the typical challenges for the two tasks, such as the inaccuracy of weak shadows, complicated shape shadow detection, and the style inconsistency

CRediT authorship contribution statement

Jin Tang: Supervision, Project administration. Qing Luo: Conceptualization, Methodology, Software, Writing - original draft. Fan Guo: Writing - original draft, Writing - review & editing. Zhihu Wu: Visualization, Investigation. Xiaoming Xiao: Data curation, Investigation. Yan Gao: Resources, Validation.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61502537, and the Natural Science Foundation of Hunan Province of China under Grant 2018JJ3681, 2016JJ2150.

References (31)

JiangC. et al.
Shadow segmentation and classification in a constrained environment
CVGIP: Image Underst.
(1994)
SalvadorE. et al.
Cast shadow segmentation using invariant color features
Comput. Vis. Image Underst.
(2004)
CucchiaraR. et al.
Detecting moving objects, ghosts, and shadows in video streams
IEEE Trans. Pattern Anal. Mach. Intell.
(2003)
NadimiS. et al.
Physical models for moving shadow and object detection in video
IEEE Trans. Pattern Anal. Mach. Intell.
(2004)
A. Yoneyama, C.H. Yeh, C.C.J. Kuo, Moving cast shadow elimination for robust vehicle extraction based on 2D joint...
E. Salvador, A. Cavallaro, T. Ebrahimi, Shadow identification and classification using invariant color models, in: 2001...
J. Zhu, K.G. Samuel, S.Z. Masood, et al. Learning to recognize shadows in monochromatic natural images, in: 2010 IEEE...
J.-F. Lalonde, A.A. Efros, S.G. Narasimhan, Detecting ground shadows in outdoor consumer photographs, in: European...
M. Baba, Asada N., Shadow removal from a real picture, in: Siggraph Conference on Sketches & Applications: in...
E. Arbel, Hel-Or H., Texture-preserving shadow removal in color images containing curved surfaces, in: 2007 IEEE...

F. Liu, M. Gleicher, Texture-consistent shadow removal, in: Computer Vision - ECCV 2008, Pt Iv, Proceedings, 2008, pp....

FinlaysonG.D. et al.

Entropy minimization for shadow removal

Int. J. Comput. Vis.

(2009)

YangQ. et al.

Shadow removal using bilateral filtering

IEEE Trans. Image Process.

(2012)

GuoR. et al.

Paired regions for shadow detection and removal

IEEE Trans. Pattern Anal. Mach. Intell.

(2013)

H. Gong, D. Cosker, Interactive shadow editing from single images, in: C.V. Jawahar, S. Shan (Eds.), Computer Vision -...

Cited by (9)

SEAT-YOLO: A squeeze-excite and spatial attentive you only look once architecture for shadow detection
2023, Optik
Recently, several deep learning-based architectures have been utilized for the recognition of shadow regions in images and videos with the aim of classification and segmentation of shadow instances. The present work aims towards solving the problem of shadow recognition as a detection and regression problem. To solve the problem of shadow detection in images and videos, we have developed SEAT-YOLO, a deep learning-based shadow detection architecture. The backbone network of the proposed architecture is based on convolution layers, squeeze-excite blocks, spatial attention module, and spatial pyramid pooling layer and the detection network utilizes YOLO detection heads. The proposed SEAT-YOLO architecture is trained and tested on the publicly available SBU shadow detection dataset. For shadow detection on the SBU dataset, it achieved a mAP value of 59.53 %. Furthermore, the proposed SEAT-YOLO is capable of detecting small shadow regions with high accuracy which is a challenge with most deep learning-based detection architectures. Moreover, the other advantage of the proposed SEAT-YOLO is its ability to detect multiple shadow regions in a single image by drawing precise bounding boxes for overlapping shadows. The proposed SEAT-YOLO architecture has high practical implications in driverless vehicles for detection of shadow regions and take effective driving decisions.
Rapid -YOLO: A novel YOLO based architecture for shadow detection
2022, Optik
Shadow detection is widely considered as segmentation problem that happens to be time consuming process with lots of duplication of results. The alternative to segmentation can be the applicability of single stage detectors like YOLO (You Only Look Once) that can run on the input image once and perform the detections during single pass. The study proposes a novel deep learning-based architecture namely, Rapid-YOLO which is an extended form of YOLOv3 architecture for detection of shadows. The proposed model is an extension of YOLOv3 architecture with addition of extra YOLO detection layers and convolution layers for aiding detection accuracy. The model exploits use of spatial pyramid pooling for scaling variations, a novel non-maximum suppression using chaotic whale optimization (CW-NMS) for overpowering false bounding box detections. We further propose tangent loss function for confidence and classification loss to improve training time and reduce errors. Drop block regularization is also used to reduce over-fitting and optimize results for shadow detection. Further, lack of proper dataset for single shot detectors encouraged us to create a novel dataset for the use of YOLO based detectors to boost further work in this direction. The model has been verified by rich experiments and results prove that the proposed methodology is superior to the state-of-the-art methods in terms of objective quantization and in subjective vision.
Brightness-gradient difference feature guided shadow removal method
2022, Knowledge-Based Systems
Citation Excerpt :
The first global network extracts shadow-related information and the second realizes shadow removal based on shadow-related information. In addition, Hu [37], Liu [38], Wang [39] and Tang [40] build Mask-ShadowGAN, Lightness-Guided Network, STackedCGAN and SDRNet to provide the results of shadow removal, respectively. After analyzing shadows in the image, Le [14] designs a shadow decomposition model, then uses a convolutional neural network to separate shadows from the outdoor image, and finally designs a shadow removal network to remove the separated shadows.
Shadow removal is a fundamental and pivotal task to build the high-level cognition in the computer vision field. Due to the fact that the existing shadow removal methods cannot effectively remove shadows from the outdoor image and deal with shadow boundaries, we construct a convolutional neural network without the process of shadow detection for the shadow removal task. The constructed CNN avoids the risk of gradient vanishing by designing double-attention residual block and improves the performance of shadow removal by fusing the knowledge transfer idea. Specially, we design a hand-crafted feature, named brightness-gradient difference feature, to distinguish shadow boundary pixels from non-shadow boundary pixels, and the designed feature is fused into the loss function to dilute or even eliminate the existing shadow boundaries. Extensive experiments using three public shadow removal benchmarks with three measurable indicators are reported in this paper. The results of experiments demonstrate that the proposed method has an effective performance for the shadow removal task. The ablation studies validate the structural rationality of the proposed method.
Hyperspectral image-based vegetation index (HSVI): A new vegetation index for urban ecological research
2021, International Journal of Applied Earth Observation and Geoinformation
Citation Excerpt :
As cities expand, so does the number of high-rise towers and the shadowed area (Liu et al., 2011; Mostafa, 2017; Shahtahmassebi et al., 2013). However, since there is extremely little optical information in shadows, extracting shadowed vegetation is difficult (Sasi and Govindan, 2016; Tang et al., 2020; Wu et al., 2021). The emergence of new materials in cities, such as artificial sports surfaces in stadiums and blue roofs, also presents significant obstacles to vegetation extraction techniques (Guo and Li, 2020; Heiden et al., 2007; Herold et al., 2004).
Accurately monitoring the quantity and quality of urban vegetation contributes to regional greening efforts and improves the understanding of vegetation's impact on the environment. However, factors such as building shadows and synthetic materials can greatly obstruct vegetation estimates. Additionally, vegetation indices (VIs) saturate quickly in high biomass conditions, complicating vegetation quality assessments. To address these issues, we propose a new vegetation index, namely the hyperspectral image-based vegetation index (HSVI). HSVI is built in three steps: band selection, saturable band reconstruction, and index structure redefinition. Firstly, we select four representative bands and construct an enhanced vegetation index (EVI) to eliminate the interference of complex urban surface factors. Secondly, we reconstruct the easily saturable band (760 nm) through an exponential function to form an optimized enhanced vegetation index (OEVI). Finally, the index structure is redefined by adding the denominator of the sum of the red edge (689 nm) and green (520 nm) bands to OEVI to further enhance the spectral information of vegetation and eliminate the saturation problem typical of VIs. Three datasets with different geomorphological features (Shanghai Theatre Academy, Dazhu Mountain, and the University of Houston) are used to compare the performance of HSVI with six VIs widely adopted in urban ecological research, i.e., the difference vegetation index (DVI), the normalized difference vegetation index (NDVI), the simple ratio (SR), the optimized soil-adjusted vegetation index (OSVAI), the modified transformed vegetation index (MTVI2) and the wide-dynamic-range vegetation index (WDRVI). The results show that the vegetation extraction accuracy of HSVI is more than 90%, which is significantly higher than those of the other VIs. Additionally, HSVI also solves the VIs saturation issue. Therefore, HSVI shows high potential usefulness for urban ecological research applications.
RDNet: Regression Dense and Attention for Object Detection in Traffic Symbols
2021, IEEE Sensors Journal
Shadow detection from images using fuzzy logic and PCPerturNet
2021, IET Image Processing

View all citing articles on Scopus

^☆: No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.image.2020.115832.

View full text

SDRNet: An end-to-end shadow detection and removal network☆

Highlights

Abstract

Introduction

Section snippets

Related works

SDRNet network structure

Experimental results and analysis

Discussion

Conclusion

CRediT authorship contribution statement

Acknowledgements

CVGIP: Image Underst.

Comput. Vis. Image Underst.

Detecting moving objects, ghosts, and shadows in video streams

IEEE Trans. Pattern Anal. Mach. Intell.

Physical models for moving shadow and object detection in video

IEEE Trans. Pattern Anal. Mach. Intell.

Entropy minimization for shadow removal

Int. J. Comput. Vis.

Shadow removal using bilateral filtering

IEEE Trans. Image Process.

Paired regions for shadow detection and removal

IEEE Trans. Pattern Anal. Mach. Intell.