This paper proposes a novel spatial–temporal deep neural network (STDNN) that is applied to multi-view gait recognition. The STDNN comprises a temporal feature network (TFN) and a spatial feature network (SFN). In TFN, a feature sub-network is adopted to extract the low-level edge features of gait silhouettes. These features are input to the spatial–temporal gradient (STG) network that adopts a STG unit and a long short-term memory unit to extract the STG features. In SFN, the spatial features of gait sequences are extracted by multilayer convolutional neural networks from a gait energy image. The SFN is optimized by classification loss and verification loss jointly, which makes inter-class variations larger than intra-class variations. After training, the TFN and the SFN are employed to extract temporal and spatial features, respectively, which are applied to multi-view gait recognition. Finally, the combined predicted probability is adopted to identify individuals by the differences in their gaits. To evaluate the performance of the STDNN, extensive evaluations are carried out based on the CASIA-B, OU-ISIR, and CMU MoBo data sets. The best recognition scores achieved by STDNN are 95.67% under an identical view, 93.64% under a cross-view, and 92.54% under a multi-view. State-of-the-art approaches are compared with the STDNN in various situations. The results show that the STDNN outperforms the other methods and demonstrates the great potential of the STDNN for practical applications in the future.

中文翻译：

---

基于时空深度神经网络的多视角步态识别

本文提出了一种新的时空深度神经网络（STDNN），用于多视角步态识别。STDNN 包括时间特征网络（TFN）和空间特征网络（SFN）。在TFN中，采用特征子网络来提取步态轮廓的低级边缘特征。这些特征被输入到时空梯度（STG）网络中，该网络采用一个 STG 单元和一个长短期记忆单元来提取 STG 特征。在SFN中，步态序列的空间特征是通过多层卷积神经网络从步态能量图像中提取的。SFN 通过分类损失和验证损失联合优化，这使得类间变化大于类内变化。训练后，分别使用 TFN 和 SFN 提取时间和空间特征，应用于多视角步态识别。最后，采用组合预测概率通过步态差异来识别个体。为了评估 STDNN 的性能，基于 CASIA-B、OU-ISIR 和 CMU MoBo 数据集进行了广泛的评估。STDNN 获得的最佳识别分数在相同视图下为 95.67%，在交叉视图下为 93.64%，在多视图下为 92.54%。在各种情况下将最先进的方法与 STDNN 进行比较。结果表明 STDNN 优于其他方法，并证明了 STDNN 在未来实际应用中的巨大潜力。为了评估 STDNN 的性能，基于 CASIA-B、OU-ISIR 和 CMU MoBo 数据集进行了广泛的评估。STDNN 获得的最佳识别分数在相同视图下为 95.67%，在交叉视图下为 93.64%，在多视图下为 92.54%。在各种情况下将最先进的方法与 STDNN 进行比较。结果表明 STDNN 优于其他方法，并证明了 STDNN 在未来实际应用中的巨大潜力。为了评估 STDNN 的性能，基于 CASIA-B、OU-ISIR 和 CMU MoBo 数据集进行了广泛的评估。STDNN 获得的最佳识别分数在相同视图下为 95.67%，在交叉视图下为 93.64%，在多视图下为 92.54%。在各种情况下将最先进的方法与 STDNN 进行比较。结果表明 STDNN 优于其他方法，并证明了 STDNN 在未来实际应用中的巨大潜力。