Skip to main content
SpringerLink
Log in

Residual connection-based graph convolutional neural networks for gait recognition

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The walking manner of a person, also known as gait, is a unique behavioral biometric trait. Existing methods for gait recognition predominantly utilize traditional machine learning. However, the performance of gait recognition can deteriorate under challenging conditions including environmental occlusion, bulky clothing, and different viewing angles. To provide an effective solution to gait recognition under these conditions, this paper proposes a novel deep learning architecture using Graph Convolutional Neural Network (GCNN) that incorporates residual connections for gait recognition from videos. The optimized feature map of the proposed GCNN architecture exhibits the invariant property to viewing angle and subject’s clothing. The residual connection is used to capture both spatial and temporal features of a gait sequence. The kinematic dependency extracted from shallower network layer is propagated to deeper layer using residual connection-based GCNN architecture. The proposed method is validated on CASIA-B gait dataset and outperforms all recent state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Xiao, Q.: Technology review-biometrics-technology, application, challenge, and computational intelligence solutions. IEEE Comput.l Intell. Mag. 2(2), 5–25 (2007)

  2. Boulgouris, N.V., Hatzinakos, D., Plataniotis, K.N.: Gait recognition: a challenging signal processing technology for biometric identification. IEEE Sig. Process. Mag. 22(6), 78–90 (2005)

    Article  Google Scholar 

  3. Ahmed, F., Paul, P.P., Gavrilova, M.L.: DTW-based kernel and rank-level fusion for 3d gait recognition using kinect. The Vis. Comput. 31(6), 915–924 (2015)

    Article  Google Scholar 

  4. Obaidat, M.S., Traore, I., Woungang, I.: Biometric-Based Physical and Cybersecurity Systems. Springer, Berlin (2019)

    Book  Google Scholar 

  5. Zhang, Y., Zheng, J., Magnenat-Thalmann, N.: Example-guided anthropometric human body modeling. The Vis. Comput. 31(12), 1615–1631 (2015)

    Article  Google Scholar 

  6. Seifert, A.-K., Zoubir, A. M., Amin, M. G.: Radar-based human gait recognition in cane-assisted walks, in 2017 IEEE Radar Conference (RadarConf), pp. 1428–1433, IEEE, (2017)

  7. Thalmann, N. M., Thalmann, D.: Modeling behaviour for social robots and virtual humans, in SIGGRAPH ASIA 2016 Courses, pp. 1–231, ACM, (2016)

  8. Rahman, M. W., Gavrilova, M. L.: Kinect gait skeletal joint feature-based person identification, in 2017 IEEE 16th international conference on cognitive informatics & Cognitive Computing (ICCI* CC), pp. 423–430, IEEE, (2017)

  9. Ahmed, F., Paul, P. P., Gavrilova, M. L.: Joint-triplet motion image and local binary pattern for 3d action recognition using kinect,’ in Proceedings of the 29th international conference on computer animation and social agents, pp. 111–119, (2016)

  10. Ahmed, F., Bari, A.H., Gavrilova, M.L.: Emotion recognition from body movement. IEEE Access 8, 11761–11781 (2019)

    Article  Google Scholar 

  11. Cho, C.-W., Chao, W.-H., Lin, S.-H., Chen, Y.-Y.: A vision-based analysis system for gait recognition in patients with Parkinson’s disease. Exp. Syst. Appl. 36(3), 7033–7039 (2009)

    Article  Google Scholar 

  12. Gavrilova, M. L., Ahmed, F., Bari, A. H., Liu, R.,Liu, T., Maret, Y., Sieu, B. K., Sudhakar, T.: Multi-modal motion-capture-based biometric systems for emergency response and patient rehabilitation, in Research Anthology on Rehabilitation Practices and Therapy, pp. 653–678, IGI global, (2021)

  13. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  14. LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. The Handbook Brain Theory Neural Netw. 3361(10), 1995 (1995)

    Google Scholar 

  15. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., Bottou, L.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. vol. 11, no. 12, (2010)

  16. Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond euclidean data. IEEE Sig. Process. Mag. 34(4), 18–42 (2017)

    Article  Google Scholar 

  17. Liao, R., Cao, C., Garcia, E. B., Yu, S., Huang, Y.: Pose-based temporal-spatial network (PTSN) for gait recognition with carrying and clothing variations, in Chinese conference on biometric recognition, pp. 474–483, Springer, (2017)

  18. He, Y., Zhang, J., Shan, H., Wang, L.: Multi-task gans for view-specific feature learning in gait recognition. IEEE Trans. Inform. Foren. Secur. 14(1), 102–113 (2018)

    Article  Google Scholar 

  19. Wu, Z., Huang, Y., Wang, L., Wang, X., Tan, T.: A comprehensive study on cross-view gait based human identification with deep CNNs. IEEE Trans. Pattern Anal. Mach. Intell. 39(2), 209–226 (2016)

    Article  Google Scholar 

  20. Zhang, Z., Tran, L., Yin, X., Atoum, Y., Liu, X., Wan, J., Wang, N.: Gait recognition via disentangled representation learning, in IEEE conference on computer vision and pattern recognition, pp. 4710–4719, (2019)

  21. Chao, H., He, Y., Zhang, J., Feng, J.: Gaitset: regarding gait as a set for cross-view gait recognition. AAAI Conf. Artif. Intell. 33, 8126–8133 (2019)

    Google Scholar 

  22. Wolf, T., Babaee, M.,Rigoll, G.: Multi-view gait recognition using 3d convolutional neural networks, in 2016 IEEE international conference on image processing (ICIP), pp. 4165–4169, IEEE, (2016)

  23. Zhang, C., Liu, W., Ma, H., Fu, H.: Siamese neural network based gait recognition for human identification, in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2832–2836, IEEE, (2016)

  24. Battistone, F., Petrosino, A.: TGLSTM: a time based graph deep learning approach to gait recognition. Pattern Recog. Lett. 126, 132–138 (2019)

    Article  Google Scholar 

  25. Bari, A.H., Gavrilova, M.L.: Artificial neural network based gait recognition using kinect sensor. IEEE Access 7, 162708–162722 (2019)

    Article  Google Scholar 

  26. Lin, B., Zhang, S., Bao, F.: Gait recognition with multiple-temporal-scale 3d convolutional neural network, in 28th ACM international conference on multimedia, pp. 3054–3062, (2020)

  27. Tu, Z., Xie, W., Qin, Q., Poppe, R., Veltkamp, R.C., Li, B., Yuan, J.: Multi-stream CNN: learning representations based on human-related regions for action recognition. Pattern Recog. 79, 32–43 (2018)

    Article  Google Scholar 

  28. Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: Openpose: realtime multi-person 2d pose estimation using part affinity fields, arXiv:1812.08008, (2018)

  29. Alp Güler, R., Neverova, N., Kokkinos, I.: Densepose: dense human pose estimation in the wild, in IEEE conference on computer vision and pattern recognition, pp. 7297–7306, 2018

  30. Toshev, A., Szegedy, C.: Deeppose: human pose estimation via deep neural networks, in IEEE conference on computer vision and pattern recognition, pp. 1653–1660, (2014)

  31. Li, N., Zhao, X., Ma, C.: A model-based gait recognition method based on gait graph convolutional networks and joints relationship pyramid mapping, arXiv:2005.08625, (2020)

  32. Shaik, S.: OpenPose based gait recognition using triplet loss architecture. PhD thesis, Dublin, National College of Ireland, (2020)

  33. Liao, R., Yu, S., An, W., Huang, Y.: A model-based gait recognition method with body pose and human prior knowledge. Pattern Recog. 98, (2020)

    Article  Google Scholar 

  34. Mao, M.,Song, Y.: Gait recognition based on 3d skeleton data and graph convolutional network, in International joint conference on biometrics, pp. 1–8, IEEE, 2020

  35. Shao, H., Zhong, D.: Few-shot palmprint recognition via graph neural networks. Electron. Lett. 55(16), 890–892 (2019)

    Article  Google Scholar 

  36. Wang, W., Lu, X., Shen, J., Crandall, D. J., Shao, L.: Zero-shot video object segmentation via attentive graph neural networks, in ieee international conference on computer vision, pp. 9236–9245, (2019)

  37. Valsesia, D., Fracastoro, G.,Magli, E.: Image denoising with graph-convolutional neural networks, in 2019 IEEE international conference on image processing (ICIP), pp. 2399–2403, IEEE, (2019)

  38. Zhang, W., Lin, Z., Cheng, J., Ma, C., Deng, X., Wang, H.: STA-GCN: two-stream graph convolutional network with spatial-temporal attention for hand gesture recognition. The Vis. Comput. 36(10), 2433–2444 (2020)

    Article  Google Scholar 

  39. Shen, Y., Li, H., Yi, S., Chen, D., Wang, X.: Person re-identification with deep similarity-guided graph neural network, in European conference on computer vision (ECCV), pp. 486–504, 2018

  40. Zhang, J., Ye, G., Tu, Z., Qin, Y., Zhang, J., Liu, X., Luo, S.: A spatial attentive and temporal dilated. SATD) gcn for skeleton-based action recognition, CAAI Transactions on Intelligence Technology (2020)

    Google Scholar 

  41. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with directed graph neural networks, in IEEE conference on computer vision and pattern recognition, pp. 7912–7921, (2019)

  42. Ye, F., Pu, S., Zhong, Q., Li, C., Xie, D., Tang, H.: Dynamic GCN: context-enriched topology learning for skeleton-based action recognition, in 28th ACM international conference on multimedia, pp. 55–63, (2020)

  43. Huang, B., Carley, K. M.: Residual or gate? towards deeper graph neural networks for inductive graph representation learning, arXiv:1904.08035, (2019)

  44. Murray, M.P.: Gait as a total pattern of movement: including a bibliography on gait. Am. J. Phys. Med. Rehab. 46(1), 290–333 (1967)

    Google Scholar 

  45. Murray, M.P., Drought, A.B., Kory, R.C.: Walking patterns of normal men. JBJS 46(2), 335–360 (1964)

    Google Scholar 

  46. BenAbdelkader, C., Cutler, R.G., Davis, L.S.: Gait recognition using image self-similarity. EURASIP J. Adv. Sig. Process. 2004(4), (2004)

    Article  Google Scholar 

  47. Kipf, T. N., Welling, M.: Semi-supervised classification with graph convolutional networks, arXiv:1609.02907, (2016)

  48. Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs, arXiv:1312.6203, (2013)

  49. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, in IEEE conference on computer vision and pattern recognition, pp. 770–778, (2016)

  50. Pouyanfar, S., Wang, T., Chen, S.-C.: Residual attention-based fusion for video classification, in IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0–0, (2019)

  51. Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, Coursera: neural networks for machine learning

  52. Teepe, T., Khan, A., Gilg, J., Herzog, F., Hörmann, S., Rigoll, G.: GaitGraph: graph convolutional network for skeleton-based gait recognition, arXiv:2101.11228, (2021)

Download references

Acknowledgements

The authors acknowledge the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant funding, as well as the NSERC Strategic Partnership Grant (SPG) and the Innovation for Defense Excellence and Security Network (IDEaS) for the partial funding of this project.

Author information

Authors and Affiliations

  1. University of Calgary, Calgary, Alberta, T2N 1N4, Canada

    Md Shopon, A. S. M. Hossain Bari & Marina L. Gavrilova

Authors
  1. Md Shopon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. S. M. Hossain Bari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marina L. Gavrilova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md Shopon.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shopon, M., Bari, A.S.M.H. & Gavrilova, M.L. Residual connection-based graph convolutional neural networks for gait recognition. Vis Comput 37, 2713–2724 (2021). https://doi.org/10.1007/s00371-021-02245-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02245-9

Keywords

Search

Navigation