Human action recognition in videos is still an important while challenging task. Existing methods based on RGB image or optical flow are easily affected by clutters and ambiguous backgrounds. In this paper, we propose a novel Pose-Guided Inflated 3D ConvNet framework (PI3D) to address this issue. First, we design a spatial–temporal pose module, which provides essential clues for the Inflated 3D ConvNet (I3D). The pose module consists of pose estimation and pose-based action recognition. Second, for multi-person estimation task, the introduced pose estimation network can determine the action most relevant to the action category. Third, we propose a hierarchical pose-based network to learn the spatial–temporal features of human pose. Moreover, the pose-based network and I3D network are fused at the last convolutional layer without loss of performance. Finally, the experimental results on four data sets (HMDB-51, SYSU 3D, JHMDB and Sub-JHMDB) demonstrate that the proposed PI3D framework outperforms the existing methods on human action recognition. This work also shows that posture cues significantly improve the performance of I3D.

在挑战性任务中，视频中的人类动作识别仍然是重要的。现有的基于RGB图像或光流的方法很容易受到杂波和模糊背景的影响。在本文中，我们提出了一种新颖的姿势引导式膨胀3D ConvNet框架（PI3D）以解决此问题。首先，我们设计一个时空姿势模块，为充气3D ConvNet（I3D）提供必要的线索。姿势模块包括姿势估计和基于姿势的动作识别。其次，对于多人估计任务，引入的姿势估计网络可以确定与动作类别最相关的动作。第三，我们提出了一个基于分层姿势的网络来学习人类姿势的时空特征。此外，基于姿势的网络和I3D网络在最后的卷积层融合在一起，而不会损失性能。最后，在四个数据集（HMDB-51，SYSU 3D，JHMDB和Sub-JHMDB）上的实验结果表明，所提出的PI3D框架优于现有的人类动作识别方法。这项工作还表明，姿势提示可以显着提高I3D的性能。