Abstract
With the increasing population, the probability of occurrence of different kinds of crowd anomalies gets frequent. Blockage on roads, the lighting condition, and the uneven movement of humans and vehicles makes it a tough and challenging problem. The paper proposes the combined use of a convolutional neural network and bidirectional LSTM to solve the task. CNN helps extract frame-level features of the optical flow over the video evaluated by the Lucas Kanade algorithm. A novel approach of improving the predicted class with the domain knowledge of datasets is also performed. The proposed methodology is tested on the crowd anomaly dataset's benchmark datasets, namely UCSD Ped-1 and UCSD Ped-2, and it outperforms various other existing state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Hettiarachchi, A.L., Thathsarani, H.O., Wickramasinghe, P.U., Wickramasuriya, D.S., Rodrigo, R.: Extensible video surveillance software with simultaneous event detection for low and high density crowd analysis. In: 7th International Conference on Information and Automation for Sustainability, Colombo, Sri Lanka, pp. 1–6 (2014). https://doi.org/10.1109/ICIAFS.2014.7069590
Piciarelli, C., Micheloni, C., Foresti, G.L.: Trajectory-based anomalous event detection. IEEE Trans. Circuits Syst. Video Technol. (2008). https://doi.org/10.1109/TCSVT.2008.2005599
Tung, F., Zelek, J.S., Clausi, D.A.: Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis. Comput. (2011). https://doi.org/10.1016/j.imavis.2010.11.003
Sabokrou, M., Fayyaz, M., Fathy, M., Klette, R.: Deep-cascade: cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans. Image Process. (2017). https://doi.org/10.1109/TIP.2017.2670780
Xie, S., Zhang, X., Cai, J.: Video crowd detection and abnormal behavior model detection based on machine learning method. Neural Comput. Appl. (2019). https://doi.org/10.1007/s00521-018-3692-x
Ojha, N., Vaish, A.: Spatiooral anomaly detection in crowd movement using SIFT. In: 2nd International Conference on Inventive Systems and Control (ICISC), Coimbatore, India, pp. 646–654 (2018). https://doi.org/10.1109/ICISC.2018.8398878
Singh, K., Rajora, S., Vishwakarma, D.K., Tripathi, G., Kumar, S., Walia, G.S.: Crowd anomaly detection using aggregation of ensembles of fine-tuned ConvNets. Neurocomputing 371, 188–198 (2020). https://doi.org/10.1016/j.neucom.2019.08.059
Zhuang, N., Ye, J., Hua, K.A.: Convolutional DLSTM for crowd scene understanding. In: Proc.—2017 IEEE Int. Symp. Multimedia, ISM 2017, vol. 2017-Janua, pp. 61–68. https://doi.org/10.1109/ISM.2017.19 (2017)
Dhole, H., Sutaone, M., Vyas, V.: Anomaly detection using convolutional spatiotemporal autoencoder. In: 2019 10th Int. Conf. Comput. Commun. Netw. Technol. ICCCNT 2019, pp. 1–5. https://doi.org/10.1109/ICCCNT45670.2019.8944523 (2019)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. (1997). https://doi.org/10.1109/78.650093
Feng, Y., Yuan, Y., Lu, X.: Learning deep event models for crowd anomaly detection. Neurocomputing (2017). https://doi.org/10.1016/j.neucom.2016.09.063
Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. In: British Machine Vision Conference (BMVC), pp 8.1–8.12 (2015). https://doi.org/10.5244/c.29.8
Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE Trans. Pattern Anal. Mach. Intell. (2014). https://doi.org/10.1109/TPAMI.2013.111
Xu, D., Song, R., Wu, X., Li, N., Feng, W., Qian, H.: Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts. Neurocomputing 143, 144–152 (2014). https://doi.org/10.1016/j.neucom.2014.06.011
Cong, Y., Yuan, J., Tang, Y.: Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans. Inf. Forensics Secur. 8(10), 1590–1599 (2013). https://doi.org/10.1109/TIFS.2013.2272243
Revathi, A.R., Kumar, D.: An efficient system for anomaly detection using deep learning classifier. Signal Image Video Process. 11(2), 291–299 (2017). https://doi.org/10.1007/s11760-016-0935-0
Zhou, J.T., Du, J., Zhu, H., Peng, X., Liu, Y., Goh, R.S.M.: AnomalyNet: an anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. (2019). https://doi.org/10.1109/TIFS.2019.2900907
Chu, W., Xue, H., Yao, C., Cai, D.: Sparse coding guided spatiotemporal feature learning for abnormal event detection in large videos. IEEE Trans. Multimed. 21(1), 246–255 (2019). https://doi.org/10.1109/TMM.2018.2846411
Khan, M.U.K., Park, H.S., Kyung, C.M.: Rejecting motion outliers for efficient crowd anomaly detection. IEEE Trans. Inf. Forensics Secur. 14(2), 541–556 (2018). https://doi.org/10.1109/TIFS.2018.2856189
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sabih, M., Vishwakarma, D.K. Crowd anomaly detection with LSTMs using optical features and domain knowledge for improved inferring. Vis Comput 38, 1719–1730 (2022). https://doi.org/10.1007/s00371-021-02100-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-021-02100-x