This paper studies robust gait features against pedestrian carrying and clothing condition changes. Inspired by the fact that humans pay more attention to pose details based on part movements when completing a gait recognition task, we introduce pose information into the convolutional network without complex computation of human modeling. We construct a multimodal set-based deep convolutional network (mmGaitSet). The mmGaitSet consists of two independent feature extractors which extract the body features from silhouettes and the part features from pose heatmaps, respectively. Joint training of two feature extractors make them complement each other. We combine intra-modal fusion and inter-modal fusion into the network. The intra-modal fusion integrates the low-level structural features and high-level semantic features, to improve the discrimination of single modality features. The inter-modal fusion fully aggregates the complementary information between different modalities to enhance the pedestrian gait presentation. The state-of-the-art results are achieved on the challenging CASIA-B dataset outperforming recent competing methods, with up to 92.5% and 80.3% average rank-1 accuracies under bag-carrying and coat-wearing walking conditions, respectively.

本文研究了针对行人携带和服装条件变化的稳健步态特征。受人类在完成步态识别任务时更关注基于部位运动的姿势细节这一事实的启发，我们将姿势信息引入卷积网络，而无需对人体建模进行复杂的计算。我们构建了一个基于多模态集的深度卷积网络（mmGaitSet）。mmGaitSet 由两个独立的特征提取器组成，分别从轮廓中提取身体特征，从姿势热图中提取部分特征。两个特征提取器的联合训练使它们相得益彰。我们将模内融合和模间融合结合到网络中。模内融合融合了低级结构特征和高级语义特征，提高对单一模态特征的辨别能力。模态间融合充分聚合了不同模态之间的互补信息，以增强行人步态表现。在具有挑战性的 CASIA-B 数据集上取得了最先进的结果，优于最近的竞争方法，在背袋和穿着外套的步行条件下，平均 rank-1 准确率分别高达 92.5% 和 80.3%。