Skip to main content
Log in

Real-time crowd behavior recognition in surveillance videos based on deep learning methods

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Automatic video surveillance in public crowded places has been an active research area for security purposes. Traditional approaches try to solve the crowd behavior recognition task using a sequential two-stage pipeline as low-level feature extraction and classification. Lately, deep learning has shown promising results in comparison to traditional methods by extracting high-level representation and solving the problem in an end-to-end pipeline. In this paper, we investigate a deep architecture for crowd event recognition to detect seven behavior categories in PETS2009 event recognition dataset. More especially, we apply an integrated handcrafted and Conv-LSTM-AE method with optical flow images as input to extract a high-level representation of data and conduct classification. After achieving a latent representation of input optical flow image sequences in the bottleneck of autoencoder(AE), the architecture is split into two separate branches, one as AE decoder and the other as the classifier. The proposed architecture is jointly trained for representation and classification by defining two different losses. The experimental results in comparison to the state-of-the-art methods demonstrate that our algorithm can be promising for real-time event recognition and achieves a better performance in calculated metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Li, T., Chang, H., Wang, M., Ni, B., Hong, R., Yan, S.: Crowded scene analysis: a survey. IEEE Trans. Circ. Syst. Video Technol. 25(3), 367–386 (2014)

    Article  Google Scholar 

  2. Khan, M.T., Ali, A., Durrani, M.Y., Siddiqui, I.: Survey of holistic crowd analysis models. J. Comput. Sci. Commun. 1(1), 1–9 (2015)

    Google Scholar 

  3. Yuan, Y., Fang, J., Wang, Q.: Online anomaly detection in crowd scenes via structure analysis. IEEE Trans. Cybern. 45(3), 548–561 (2014)

    Article  Google Scholar 

  4. Ferryman, J.: PETS 2009 benchmark data (2009). http://www.cvg.rdg.ac.uk/PETS2009/a.html

  5. Ferryman, J., Shahrokni, A.: Pets2009: dataset and challenge. In: 2009 Twelfth IEEE international workshop on performance evaluation of tracking and surveillance, pp. 1–6. IEEE (2009)

  6. Shi, J.: Good features to track. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 593–600. IEEE (1994)

  7. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  8. Tomasi, C., Kanade, T.: Detection and tracking of point features. Technical report CMU-CS-91-132, CMU Google Scholar (1991)

  9. Fradi, H., Dugelay, J.L.: Spatial and temporal variations of feature tracks for crowd behavior analysis. J. Multimodal User Interfaces 10(4), 307–317 (2016)

    Article  Google Scholar 

  10. Fradi, H., Dugelay, J.L.: Sparse feature tracking for crowd change detection and event recognition. In: 22nd International Conference on Pattern Recognition, pp. 4116–4121. IEEE (2014)

  11. Rao, A.S., Gubbi, J., Palaniswami, M.: An improved approach to crowd event detection by reducing data dimensions. In: Advances in Signal Processing and Intelligent Recognition Systems, pp. 85–96. Springer, Cham (2016)

  12. Wang, L., Qiao, Y., Tang, X.: Action recognition with trajectory-pooled deep-convolutional descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4305–4314 (2015)

  13. Zhang, W., Hou, Y., Wang, S.: Event recognition of crowd video using corner optical flow and convolutional neural network. In: 8th International Conference on Digital Image Processing (ICDIP 2016), vol. 10033, p. 100335K. International Society for Optics and Photonics (August)

  14. Chan, A.B., Morrow, M., Vasconcelos, N.: Analysis of crowded scenes using holistic properties. In: Performance Evaluation of Tracking and Surveillance workshop at CVPR, pp. 101–108 (2009)

  15. Cermeno, E., Mallor, S., Sigüenza, J.A.: Learning crowd behavior for event recognition. In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS), pp. 1–5. IEEE (2013)

  16. Benabbas, Y., Ihaddadene, N., Djeraba, C.: Motion pattern extraction and event detection for automatic visual surveillance. EURASIP J. Image Video Process. 2011(1), 163682 (2011)

    Google Scholar 

  17. Briassouli, A., Kompatsiaris, I.: Spatiotemporally localized new event detection in crowds. In: IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 928–933. IEEE (2011)

  18. Grigg, O.A., Farewell, V.T., Spiegelhalter, D.J.: Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts. Stat. Methods Med. Res. 12(2), 147–170 (2003)

    Article  MathSciNet  Google Scholar 

  19. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–942. IEEE (2009)

  20. Mehran, R., Moore, B.E., Shah, M.: A streakline representation of flow in crowded scenes. In: European Conference on Computer Vision, pp. 439–452. Springer, Berlin (2010)

  21. Huang, S., Huang, D., Khuhro, M.A.: Crowd motion analysis based on social force graph with streak flow attribute. J. Electr. Comput. Eng. 2015, 52 (2015)

    Google Scholar 

  22. Dee, H.M., Caplier, A.: Crowd behaviour analysis using histograms of motion direction. In: IEEE International Conference on Image Processing, pp. 1545–1548. IEEE (2010)

  23. Mousavi, H., Mohammadi, S., Perina, A., Chellali, R., Murino, V. Analyzing tracklets for the detection of abnormal crowd behavior. In: IEEE Winter Conference on Applications of Computer Vision, pp. 148–155. IEEE (2015)

  24. Wang, X., He, Z., Sun, R., You, L., Hu, J., Zhang, J.: A crowd behavior identification method combining the streakline with the high-accurate variational optical flow model. IEEE Access 7, 114572–114581 (2019)

    Article  Google Scholar 

  25. Thida, M., Eng, H.L., Monekosso, D.N., Remagnino, P.: Learning video manifolds for content analysis of crowded scenes. IPSJ Trans. Comput. Vis. Appl. 4, 71–77 (2012)

    Article  Google Scholar 

  26. Ghodsi, A.: Dimensionality reduction a short tutorial, vol 37, p 38. Department of Statistics and Actuarial Science, Univ. of Waterloo, Ontario (2006)

  27. Zhang, Y., Huang, Q., Qin, L., Zhao, S., Yao, H., Xu, P.: Representing dense crowd patterns using bag of trajectory graphs. Signal Image Video Process 8(1), 173–181 (2014)

    Article  Google Scholar 

  28. Fradi, H., Luvison, B., Pham, Q.C.: Crowd behavior analysis using local mid-level visual descriptors. IEEE Trans. Circ. Syst. Video Technol. 27(3), 589–602 (2016)

    Article  Google Scholar 

  29. Rao, A.S., Gubbi, J., Marusic, S., Palaniswami, M.: Crowd event detection on optical flow manifolds. IEEE Trans. Cybern. 46(7), 1524–1537 (2015)

    Article  Google Scholar 

  30. Su, H., Yang, H., Zheng, S., Fan, Y., Wei, S.: The large-scale crowd behavior perception based on spatio-temporal viscous fluid field. IEEE Trans. Inf. Forensics Secur. 8(10), 1575–1589 (2013)

    Article  Google Scholar 

  31. Solmaz, B., Moore, B.E., Shah, M.: Identifying behaviors in crowd scenes using stability analysis for dynamical systems. IEEE Trans. Pattern Anal. Mach. Intell. 34(10), 2064–2070 (2012)

    Article  Google Scholar 

  32. Khokher, M. R., Bouzerdoum, A., Phung, S.L.: Crowd behavior recognition using dense trajectories. In: International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7. IEEE (2014)

  33. Shuaibu, A.N., Faye, I., Ali, Y.S., Kamel, N., Saad, M.N., Malik, A.S.: Sparse representation for crowd attributes recognition. IEEE Access 5, 10422–10433 (2017)

    Article  Google Scholar 

  34. Pathan, S.S., Al-Hamadi, A., Michaelis, B.: Crowd behavior detection by statistical modeling of motion patterns. In: International Conference of Soft Computing and Pattern Recognition, pp. 81–86. IEEE (2010)

  35. Hu, X., Hu, S., Huang, Y., Zhang, H., Wu, H.: Video anomaly detection using deep incremental slow feature analysis network. IET Comput. Vis. 10(4), 258–267 (2016)

    Article  Google Scholar 

  36. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725–1732 (2014)

  37. Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2012)

    Article  Google Scholar 

  38. Wei, H., Xiao, Y., Li, R., Liu, X.: Crowd abnormal detection using two-stream Fully Convolutional Neural Networks. In: 10th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), pp. 332–336. IEEE (2018)

  39. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, pp. 568–576 (2014)

  40. Xu, Y., Lu, L., Xu, Z., He, J., Wang, J., Huang, J., Lu, J.: Towards intelligent crowd behavior understanding through the STFD descriptor exploration. Sens. Imaging 19(1), 17 (2018)

    Article  Google Scholar 

  41. Fang, Z., Fei, F., Fang, Y., Lee, C., Xiong, N., Shu, L., Chen, S.: Abnormal event detection in crowded scenes based on deep learning. Multimed. Tools Appl. 75(22), 14617–14639 (2016)

    Article  Google Scholar 

  42. Khan, G., Farooq, M.A., Hussain, J., Tariq, Z., Khan, M.U.G.: Categorization of crowd varieties using deep concurrent convolution neural network. In: 2nd International Conference on Advancements in Computational Sciences (ICACS), pp. 1–6. IEEE (2019)

  43. Burney, A., Syed, T.Q.: Crowd video classification using convolutional neural networks. In: International Conference on Frontiers of Information Technology (FIT), pp. 247–251. IEEE (2016)

  44. Borja-Borja, L.F., Saval-Calvo, M., Azorin-Lopez, J.: A short review of deep learning methods for understanding group and crowd activities. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)

  45. Li, P., Jiang, X., Sun, T., Xu, K.: Crowded scene understanding algorithm based on two-stream residual network. In: 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–6. IEEE (2017)

  46. Roli, F., Giacinto, G., Vernazza, G.: Comparison and combination of statistical and neural network algorithms for remote-sensing image classification. In: Neurocomputation in remote sensing data analysis, pp. 117–124. Springer, Berlin (1997)

  47. Wang, C., Zhao, X., Shou, Z., Zhou, Y., Liu, Y.: A discriminative tracklets representation for crowd analysis. In: IEEE International Conference on Image Processing (ICIP), pp. 1805–1809. IEEE (2015)

  48. Li, Y.: A deep spatiotemporal perspective for understanding crowd behavior. IEEE Trans. Multimed. 20(12), 3289–3297 (2018)

    Article  Google Scholar 

  49. Zhuang, N., Ye, J., Hua, K.A.: Convolutional DLSTM for crowd scene understanding. In: IEEE International Symposium on Multimedia (ISM), pp. 61–68. IEEE (2017)

  50. Erfani, S.M., Rajasegarar, S., Karunasekera, S., Leckie, C.: High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning. Pattern Recogn. 58, 121–134 (2016)

    Article  Google Scholar 

  51. Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: International Symposium on Neural Networks, pp. 189–196. Springer, Cham (2017)

  52. Fernández-Ramírez, J., Á lvarez-Meza, A., Pereira, E.M., Orozco-Gutiérrez, A., Castellanos-Dominguez, G.: Video-based social behavior recognition based on kernel relevance analysis. Vis. Comput. 36(8), 1535–1547 (2020)

  53. Deng, C., Kang, X., Zhu, Z., Wu, S.: Behavior recognition based on category subspace in crowded videos. IEEE Access 8, 222599–222610 (2020)

    Article  Google Scholar 

  54. Varghese, E., Thampi, S.M., Berretti, S.: A psychologically inspired fuzzy cognitive deep learning framework to predict crowd behavior. In: IEEE Transactions on Affective Computing (2020)

  55. Li, Q., Zhao, X., He, R., Huang, K.: Recurrent prediction with spatio-temporal attention for crowd attribute recognition. IEEE Trans. Circ. Syst. Video Technol. 30(7), 2167–2177 (2019)

    Google Scholar 

  56. Tripathi, G., Singh, K., Vishwakarma, D.K.: Convolutional neural networks for crowd behaviour analysis: a survey. Vis. Comput. 35(5), 753–776 (2019)

    Article  Google Scholar 

  57. Kiran, B.R., Thomas, D.M., Parakkal, R.: An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. J. Imaging 4(2), 36 (2018)

    Article  Google Scholar 

  58. Pérez, J.S., Meinhardt-Llopis, E., Facciolo, G.: TV-L1 optical flow estimation. Image Process. On Line 2013, 137–150 (2013)

    Article  Google Scholar 

  59. Xingjian, S.H.I., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, pp. 802–810 (2015)

  60. Yang, M., Rajasegarar, S., Erfani, S. M., Leckie, C.: Deep learning and one-class SVM based anomalous crowd detection. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)

  61. Shao, J., Kang, K., Change Loy, C., Wang, X.: Deeply learned attributes for crowded scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4657–4666 (2015)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mehran Yazdi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rezaei, F., Yazdi, M. Real-time crowd behavior recognition in surveillance videos based on deep learning methods. J Real-Time Image Proc 18, 1669–1679 (2021). https://doi.org/10.1007/s11554-021-01116-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01116-9

Keywords

Navigation