BCNet: Bidirectional collaboration network for edge-guided salient object detection

Abstract

The boundary quality is a key factor determining the success of accurate salient object detection (SOD). A number of edge-guided SOD methods have been proposed to improve the boundary quality, but shown unsatisfactory performance due to the lack of a comprehensive consideration of multi-level feature fusion and multi-type feature aggregation. To resolve this issue, we propose a novel Bidirectional Collaboration Network (BCNet), which integrates effective multi-level feature fusion and multi-type feature aggregation into a unified edge-guided SOD framework. Specifically, we first utilize multiple Consistency Saliency Maximization (CSM) modules to propagate the highest level semantic representations in a top-down progressive pathway to generate both global edge representations and a series of region representations. Multiple Bounded Feature Fusion (BFF) modules are then utilized to refine the region features with the edge features. The CSM and BFF modules enable robust multi-level feature fusion and multi-type feature aggregation with only little extra computation, which allows a high computational efficiency. Finally, BCNet is jointly trained with edge and region losses in an end-to-end manner. Extensive comparisons are conducted with 17 state-of-the-art methods on five challenging benchmark datasets. Thanks to the use of CSM and BFF modules, our BCNet outperforms existing deep learning based SOD methods, including the latest edge-guided ones, in terms of both detection accuracy and processing speed.

Introduction

Salient object detection (SOD), acting as a powerful pre-processing tool in numerous computer vision tasks, mimics the human visual attention mechanism for identifying attention-grabbing objects from natural images. It has a large number of applications, such as autonomous driving [1], robot navigation [2], visual tracking [3], image retrieval [4], aesthetics assessment [5], and content-aware image editing [6]. Inspired by the progress in perceptual psychology, early models detect salient objects using heuristic priors[7], [8] and hand-crafted features such as contrast [9] and distance transformation [10]. However, their detection performance is seriously limited in complex scenarios. Recent works have demonstrated that deep learning techniques, especially the Convolutional Neural Networks (CNNs) [11], [12], [13], [14], are particularly good at understanding visual concepts by extracting semantic features from image regions, and have achieved remarkable performance [15], [16], [17], [18]. Despite their advantages, existing methods suffer from two major limitations. First, it is still challenging to detect entire salient objects from a complex background, even using deep-based methods. Second, most existing methods are unable to accurately detect the boundaries of salient objects.

To overcome these limitations, a number of methods have been proposed in recent years [19], [20], [21], [22], [23], [24], [25], [26]. For instance, fusing the multi-level features from low-level and high-level convolutional layers [20], [21], [27] improves the detection of objects from a complex background. In addition, resorting to additional edge guidance [24], [25], [26], [28] is able to improve the accuracy of boundaries. However, most of the existing solutions only focus on addressing one aspect of the limitations, while overlooking the other one. In addition, although edge-guided methods such as [24], [28] provide encouraging boundary quality, the aggregation of multi-type features, i.e., region and edge features, is achieved by the naive concatenation or element-wise addition/multiplication, which might be suboptimal and ineffective.

To this end, we propose a novel bidirectional collaboration network, called BCNet, which integrates effective multi-level feature fusion and multi-type feature aggregation into a unified SOD framework. BCNet utilizes the edge features to guide region features, which automatically discards low-quality features and highlights more edge details, as shown in Fig. 1. Specifically, we introduce a module, called Consistency Saliency Maximization (CSM), which is inspired by the spatial attention mechanism [29], and embed it to BCNet to mitigate the discrepancy in different levels of features for effective feature fusion. Fig. 1(d) and (e) show the edge feature map and the feature map generated by fusing shallow features (b) and high-level features (c) with our CSM module. As can be observed, after fusion, the entire objects become clear and background noise is suppressed. To improve the edge sharpness and accuracy, we introduce another module, called Bounded Feature Fusion (BFF), which is inspired by the Squeeze-Excitation block in [30], to aggregate the multi-type features provided by the CSM modules. Different from existing methods relying on simple concatenation or element-wise operations [24], [28], BFF utilizes effective feature re-weighting to sharpen the edges (Fig. 1(f)). Finally, BCNet is jointly trained with edge and region losses in an end-to-end manner. It is worth noting that CSM and BFF modules enable effective multi-level feature fusion and multi-type feature aggregation, but only inducing little extra computation, which allows effective real-time processing at 52fps. The main contributions of this paper are as follows:

• We propose a novel bidirectional collaboration network BCNet for edge-guided SOD, which effectively addresses multi-level feature fusion and multi-type feature aggregation within a unified framework. Accordingly, two modules, called Consistency Saliency Maximization (CSM) and Bounded Feature Fusion (BFF), are introduced.

• We construct a new bidirectional collaboration architecture for BCNet, where local region features are first organized in a top-down progressive pathway to propagate the highest level semantic representations, and then the global edge features are used to refine the obtained region features for final prediction.

• Extensive comparisons are conducted with 17 state-of-the-art (SOTA) methods on five challenging benchmark datasets, demonstrating that the proposed BCNet performs favorably against the latest SOTA models in terms of both accuracy and speed. Notably, BCNet achieves real-time speed at 52fps, shown to be one of the fastest models comparing to SOTAs.

The rest of the paper is organized as follows. Section 2 describes related works on deep SOD, especially the edge-guided models. Section 3 describes the proposed BCNet in detail. Experimental results, performance evaluation and comparisons are presented in Section 4. Finally, conclusions are drawn in Section 5.