Human motion recognition has become an active research area in the field of computer vision due to its wide range of implementations in domains of video monitoring, virtual reality, human–machine interaction. Dealing with the problem that the RGB images cannot provide enough depth information, a multi-modal depth neural network based on joint cost function is proposed for human motion recognition. In the architecture, the features of the RGB video frames are extracted by the 3D CNN architecture while the characteristics of human motion recognition in the SSDDI graphics utilizing depth map are extracted by the LSTM. Moreover, the model utilizes joint cost function including the cross-entropy loss and the distance constraint between the feature space of training samples and their center values within each category. The experimental results on the MSR Action 3D datasets suggest that the proposed model demonstrates a higher accuracy rate than do the other competing models.

人体运动识别由于在视频监控，虚拟现实，人机交互等领域的广泛实现而成为计算机视觉领域的活跃研究领域。针对RGB图像不能提供足够的深度信息的问题，提出了一种基于联合代价函数的多模态深度神经网络用于人体运动识别。在该架构中，通过3D CNN架构提取RGB视频帧的特征，而通过LSTM提取利用深度图在SSDDI图形中进行人体运动识别的特征。此外，该模型利用联合成本函数，包括交叉熵损失和训练样本的特征空间与其每个类别内的中心值之间的距离约束。