Semantic segmentation based on the complementary information from RGB and depth images has recently gained great popularity, but due to the difference between RGB and depth maps, how to effectively use RGB-D information is still a problem. In this paper, we propose a novel RGB-D semantic segmentation network named RAFNet, which can selectively gather features from the RGB and depth information. Specifically, we construct an architecture with three parallel branches and propose several complementary attention modules. This structure enables a fusion branch and we add the Bi-directional Multi-step Propagation (BMP) strategy to it, which can not only retain the feature streams of the original RGB and depth branches but also fully utilize the feature flow of the fusion branch. There are three kinds of complementary attention modules that we have constructed. The RGB-D fusion module can effectively extract important features from the RGB and depth branch streams. The refinement module can reduce the loss of semantic information and the context aggregation module can help propagate and integrate information better. We train and evaluate our model on NYUDv2 and SUN-RGBD datasets, and prove that our model achieves state-of-the-art performances.

基于RGB和深度图像互补信息的语义分割最近大受欢迎，但由于RGB和深度图的差异，如何有效利用RGB-D信息仍然是一个问题。在本文中，我们提出了一种名为 RAFNet 的新型 RGB-D 语义分割网络，它可以选择性地从 RGB 和深度信息中收集特征。具体来说，我们构建了一个具有三个并行分支的架构，并提出了几个互补的注意力模块。这种结构启用了一个融合分支，我们在其中添加了双向多步传播（BMP）策略，既可以保留原始RGB和深度分支的特征流，又可以充分利用融合分支的特征流. 我们构建了三种互补的注意力模块。RGB-D 融合模块可以有效地从 RGB 和深度分支流中提取重要特征。细化模块可以减少语义信息的丢失，上下文聚合模块可以帮助更好地传播和整合信息。我们在 NYUDv2 和 SUN-RGBD 数据集上训练和评估我们的模型，并证明我们的模型达到了最先进的性能。