Learning multi-granularity features from multi-granularity regions for person re-identification

Abstract

Part-based methods for person re-identification have been widely studied. In existing part-based methods, although multiple parts are explored, only coarse-grained features of these parts are utilized. Thus, too much fine-grained information is discarded, which limits their ability to extract detailed discriminative features. To tackle this problem, we propose a novel person re-identification network to learn discriminative features across multiple granularities from body regions which are also multi-grained. Specifically, we detect multi-granularity body regions at different stages of a backbone network, and multi-granularity features are learned from body regions with corresponding granularities. To overcome the severe mismatching problem of fine-grained regions and to learn discriminative features, the detection of multi-granularity body regions and the learning of multi-granularity features are jointly optimized. This joint optimization pushes the learned features concentrating on body regions. Moreover, with the body regions well located, the multi-granularity features can be well aligned. Extensive experiments on four popular datasets show that our method is the state-of-the-art in recent years.

Introduction

Person re-identification (ReID) aims at identifying specific person from a set of surveillance cameras across time. It plays a significant role in many vision-related applications, e.g., video surveillance, content-based video retrieval, and identification from CCTV cameras. Compared to other computer vision tasks, ReID is of great challenge due to differences of background, deviations in shape, and occlusion of the subjects [1], [2].

Image representation learning plays a crucial role in person ReID. As shown in Fig. 1(a), images are usually fed to deep convolutional neural networks (CNNs) to extract the final representation. However, the final features are often too coarse and lose too much detail information. To solve this problem, many part-based models have been proposed [2], [3], [4], [5]. By learning discriminative local features as a complement to global features, they can extract additional rich features and thus achieve better ReID performance.

According to the local region generation way, part-based models can be divided into three categories: pose-based, attention-based, and stripe-based. In pose-based methods, prior knowledge, e.g., pose estimation or human segmentation, is used to locate local regions of a human body accurately [6], [2], [7], [8], [9]. These methods handle local regions of a human body by extra convolutional branches. The attention-based methods learn attention masks to select a focused foreground [10], [11], [12]. In stripe-based category, the feature maps are split into several predefined horizontal stripes [4], [13], [14], [15]. However, they all perform ReID with the features from the last layer, which have coarse granularity and contain limited local information. Furthermore, these methods are based on the assumption that person images are well aligned, so the corresponding stripes can be matched. However, misalignment is very common in person ReID.

However, methods in all three categories have one common drawback: though multiple parts/regions are explored, only coarse-grained features of these parts are utilized, as shown in Fig. 1(b). The local regions are first cropped either at the input [6] or at different stages in backbone CNNs [2], and are then fed into convolutional branches afterwards, leading to that the final features of these regions are coarse-grained. This limits the diversity and discrimination of the final features.

To tackle this problem, we propose to learn multi-granularity features from multi-granularity body regions for person ReID. We detect local regions across multiple granularities at different stages of a backbone network. As shown in Fig. 1(c), we detect four fine-grained body parts in the first stage, two body parts in the second stage, the whole body region in the third stage, and the whole image in the forth stage. For regions in each granularity, instead of feeding them to extra local branches afterwards, we directly apply a feature extraction module to learn corresponding features. In this manner, the original granularities of features are kept, and more detail information are retained. Finally, multi-granularity features learned from multi-granularity body regions are fused for person ReID. Fig. 1 clearly illustrates the difference between our model and existing part-based models.

It is worth noting that we do not simply conduct the multi-granularity region detection and multi-granularity feature learning in a straightforward two-stage manner. There are two challenging problems in our task. First, when fine-grained features are used, the misalignment becomes a big problem, especially for fine-grained regions, since the fine-grained features are extracted from shallow layers in which the receptive field is small and thus are very sensitive to translation, pose variations, etc. This may be the reason why current works only use coarse-grained features for all parts. Second, the fine-grained features are very sensitive to noises or other image content which are not helpful for ReID. Thus, we face the problem of how to ensure the extracted fine-grained features are discriminative for person ReID. To tackle these problems, we design a model to jointly optimize the multi-granularity region detection and multi-granularity feature learning.

In summary, the contributions of this paper are threefold:

• We learn features across multiple granularities from the backbone network without feeding them to extra local branches. In this manner, the final features are diverse: both fine-grained features with rich details and coarse-grained abstract features are well reserved.

• Our multi-granularity features are learned from multi-granularity parts. The location of multi-granularity parts and the learning of multi-granularity features are jointly optimized. This joint optimization pushes the learned features focusing on human body regions. Moreover, with the multi-granularity parts accurately located, the multi-granularity features can be well aligned, and the background noise can be well reduced.

• The proposed method achieves the best performance on four person ReID datasets. Extensive experiments on these datasets verify the effectiveness of our approach. MGRe achieves 90.1%/96.2% mAP/Rank-1 in Market1501 and 82.0%/91.3% mAP/Rank-1 in DukeMTMC-reID.