A deep multimodal feature learning network for RGB-D salient object detection

Abstract

In this paper, we propose a deep multimodal feature learning (DMFL) network for RGB-D salient object detection. The color and depth features are firstly extracted from low level to high level feature using CNN. Then the features at the high layer are shared and concatenated to construct joint feature representation of multi-modalities. The fused features are embedded to a high dimension metric space to express the salient and non-salient parts. And also a new objective function, consisting of cross-entropy and metric loss, is proposed to optimize the model. Both pixel and attribute level discriminative features are learned for semantical grouping to detect the salient objects. Experimental results show that the proposed model achieves promising performance and has about 1% to 2% improvement to conventional methods.

Introduction

Visual saliency aims to determine the parts of an image where the attention of human is attracted [1]. As depth images provide additional cues compared with color images, using depth information to detect salient objects in RGB-D images has gradually aroused people’s interest. Many efforts have been made on the issue, and the researches have been successfully applied to a broad range of practice [2], [3], [4].

Many methods has been developed to promote the advancement of artificial intelligence [5], [6], [7], [8]. Among them, CNNs are popularly used in feature extraction [9], [10]. Low-level features used in conventional saliency detection models cannot reflect high-level cues of objects, and therefore CNNs is widely adopted to locate salient objects by extracting high level information [11], [12], [13], [14], [15]. The CNN-based saliency detection models tend to learn the representation of the concatenated features of two modalities of RGB-D images. These models generally identify one pixel to be salient according to discriminative features obtained by minimizing the cross-entropy loss. With the concatenation operation, these models have limited capability of discovering the common and complementary features. With the cross-entropy loss function, these models have the limited capability of assigning homogeneous labels to pixels [16]. Therefore, on the one hand, it is necessary to find the way of effectively extracting features; on the other hand, it is necessary to describe the semantic attributes of salient and non-salient parts.

To extract those effective features, some models have been developed. As learning complementary features of multimodality separately is exhaustive, in other tasks of computer vision (for example, classification), features are commonly decomposed and distributed to specific model [17], [18]. However, this way of feature decomposition has not been developed in the task of saliency detection.

To describe the semantic attributes of regions, metric learning has been introduced in some recent works. In view of the idea of metric learning [19], [20], the samples with short distances are put together, whereas those samples with long distances are separated. It has also been introduced to the saliency detection models [21] for 2D images, in which the pixels with the similar attributes of the salient regions and non-salient regions are supposed to be salient and non-salient, respectively. However, whether an object is salient or not is determined to a certain extent by the depth where it is located. That is to say, the saliency detection model for RGB-D images should consider the distance of features from both depth and color for salient region detection. Hence, learning the metric distance of multimodal representation of features is important for RGB-D saliency detection.

In this paper, we propose a deep multimodal feature learning (DMFL) network for RGB-D salient object detection in an implicit metric space. Inspired by the work [22], we extract both features of different modalities and the sharable features from two modalities as the complementary information. Different from the traditional methods of mapping pixels to labels, we specially map the extracted features to a metric space for the model generalization. A new objective function is designed and optimized to extract both the specific features and the sharable features of two modalities by imposing a metric loss on the loss of cross-entropy. We use the metric loss regulation term to ensure that the proposed model can judge whether a pixel is salient or non-salient by a metric distance. In the experiments, it is demonstrated that the discriminative features can improve the performance of RGB-D saliency detection. In a nutshell, the contributions of this paper are summarized as follows:

• We propose a deep multimodal feature learning network for RGB-D salient object detection, which considers the attributes of pixels and regions in image-itself;

• We make the early effort to introduce metric learning for representing the implicit attribute of regions using color and depth cues;

• The features of intra-and inter-modalities are combined together to explore the complementary information for the saliency cues of multimodalities;

• Experimental results from public datasets demonstrate the effectiveness of the proposed model.

The rest of this paper is organized as follows. In Section 2, a review of the related saliency models is given. In Section 3, the proposed model is described in detail. In Section 4, the experimental results are presented and analyzed. Conclusions are provided in Section 5.