Skip to main content
SpringerLink
Log in

Eagle Eye CBVR Based on Unique Key Frame Extraction and Deep Belief Neural Network

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Great efforts have been paid to develop Content Based Video Retrieval (CBVR) due to the exponential growth of video datasets with several types of data such as visual, audio and metadata. This paper proposes Eagle Eye a new Content Based Video Retrieval Framework through applying new methods for the major three important parts of CBVR. A Subspace Clustering Algorithm is implemented with binary codes that are extracted from different videos. We introduce a Local Histogram based shot boundary detection algorithm to detect shot boundaries. Then Unique Key Frame Summarization algorithm with detected shots is applied. An integrated Local Binary Pattern, Coherence based filter and HOG algorithm is used for the Feature extraction and best features are selected and stored along with the video indices. The extracted features were then utilized to train a classifier. A Deep Belief Neural Network classifier trains each extracted feature of query frame and matches the best features of the query frames with other video features. Then we enhance the video retrieval by introducing video hashing method, combining low level and high-level semantic features for the elimination of repetitive video in the retrieval. We used precision, accuracy, sensitivity and specificity metrics to assess the applicability of the projected technique. Xiph.org and Youtube datasets are used for the valuation analysis. The experimental results show that the Eagle Eye provides better performance and less processing time compared to the other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Hu, W., Xie, N., Li, L., Zeng, X., & Maybank, S. (2011). A survey on visual content-based video indexing and retrieval. IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews), 41(6), 797–819.

    Article  Google Scholar 

  2. Bakar, Z. A., Kassim, M., Sahroni, M. N., & Anuar, N. (2017). A survey: Framework to develop retrieval algorithms of indexing techniques on learning material. Journal of Telecommunication, Electronic and Computer Engineering, 9(2–5), 43–46.

    Google Scholar 

  3. Pasumarthi, N., & Malleswari, L. (2016). An empirical study and comparative analysis of Content Based Image Retrieval (CBIR) techniques with various similarity measures

  4. Pentland, A. P., Picard, R. W., & Scarloff, S. (1994). Photobook: Tools for content-based manipulation of image databases. In Storage and retrieval for image and video databases II (vol. 2185, pp. 34–48). International Society for Optics and Photonics.

  5. Deng, Y., & Manjunath, B. (1998). NeTra-V: Toward an object-based video representation. IEEE Transactions on Circuits and Systems for Video Technology, 8(5), 616–627.

    Article  Google Scholar 

  6. Sclaroff, S., Taycher, L., & La Cascia, M. (1997). Imagerover: A content-based image browser for the world wide web. In 1997 Proceedings IEEE workshop on content-based access of image and video libraries (pp. 2–9) IEEE.

  7. Hampapur, A., et al. (1997). Virage video engine. In: Storage and retrieval for image and video databases V (vol 3022, pp. 188–199). International Society for Optics and Photonics.

  8. Beebe, C. (2000). Image indexing for multiple needs. Journal of the Art Libraries Society of North America, 19(2), 16–21.

    Google Scholar 

  9. Rui, Y., Huang, T. S., & Mehrotra, S. (1997). Content-based image retrieval with relevance feedback in MARS. In Proceedings of international conference on image processing (vol. 2, pp. 815–818). IEEE.

  10. Carson, C., Belongie, S., Greenspan, H., & Malik, J. (2002). Blobworld: Image segmentation using expectation-maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 1026–1038.

    Article  Google Scholar 

  11. Cheikh, F. A. (2004). MUVIS-a system for content-based image retrieval. Tampere University of Technology.

  12. Laaksonen, J., Koskela, M., Laakso, S., & Oja, E. (2000). PicSOM–content-based image retrieval with self-organizing maps. Pattern Recognition Letters, 21(13–14), 1199–1207.

    Article  MATH  Google Scholar 

  13. Patil, D., Potey, M. (2015). Survey of content based lecture video retrieval. International Journal of Computer Trends and Technology 19(1)

  14. Kaur, M., & Kumar, H. (2015). Keyword-based search and ranking in NPTEL lecture videos. In Proceedings of the international conference on transformations in engineering education (pp. 79–88). Springer

  15. Song, J., Gao, L., Liu, L., Zhu, X., & Sebe, N. (2018). Quantization-based hashing: a general framework for scalable image and video retrieval. Pattern Recognition, 75, 175–187.

    Article  Google Scholar 

  16. Sun, J., Liu, X., Wan, W., Li, J., Zhao, D., & Zhang, H. J. N. (2016). Video hashing based on appearance and attention features fusion via DBN. Neurocomputing, 213, 84–94.

    Article  Google Scholar 

  17. Ansari, A., & Mohammed, M. (2015). Content based video retrieval systems-methods, techniques, trends and challenges. International Journal of Computer Applications, 112(7), 13–22.

    Google Scholar 

  18. Khusro, S., Ali, Z., & Ullah, I. (2016). Recommender systems: issues, challenges, and research opportunities. In: Information science and applications (ICISA) 2016 (pp. 1179–1189). Springer

  19. Fernandez-Beltran, R., & Pla, F. (2016). Latent topics-based relevance feedback for video retrieval. Pattern Recognition, 51, 72–84.

    Article  Google Scholar 

  20. Guo, J.-M., & Prasetyo, H. (2015). Content-based image retrieval using features extracted from halftoning-based block truncation coding. IEEE Transactions on Image Processing, 24(3), 1010–1024.

    Article  MathSciNet  MATH  Google Scholar 

  21. Basu, S., Yu, Y., & Zimmermann, R. (2016). Fuzzy clustering of lecture videos based on topic modeling. In: 2016 14th international workshop on content-based multimedia indexing (CBMI) (pp. 1–6). IEEE.

  22. Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018). Self-supervised video hashing with hierarchical binary auto-encoder. IEEE Transactions on Image Processing, 27(7), 3210–3221.

    Article  MathSciNet  MATH  Google Scholar 

  23. Ye, Y., Zhao, Z., Li, Y., Chen, L., Xiao, J., & Zhuang, Y. (2017). Video question answering via attribute-augmented attention network learning. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pp. 829–832. ACM.

  24. Hauptmann, A. (2015). Fast and accurate content-based semantic search in 100M Internet Videos. In MM 2015-proceedings of the 2015 ACM multimedia conference, 2015.

  25. Deldjoo, Y., Elahi, M., Quadrana, M., & Cremonesi, P. (2018). Using visual features based on MPEG-7 and deep learning for movie recommendation. International Journal of Multimedia Information Retrieval, 7(4), 207–219.

    Article  Google Scholar 

  26. Liong, V. E., Lu, J., Tan, Y.-P., & Zhou, J. (2016). Deep video hashing. IEEE Transactions on Multimedia, 19(6), 1209–1219.

    Article  Google Scholar 

  27. Thakre, K., Rajurkar, A., & Manthalkar, R. (2016). Video partitioning and secured keyframe extraction of MPEG video. Procedia Computer Science, 78, 790–798.

    Article  Google Scholar 

  28. Guo, H., Wang, J., & Lu, H. (2016). Multiple deep features learning for object retrieval in surveillance videos. IET Computer Vision, 10(4), 268–272.

    Article  Google Scholar 

  29. Younus, Z. S., et al. (2015). Content-based image retrieval using PSO and k-means clustering algorithm. Arabian Journal of Geosciences, 8(8), 6211–6224.

    Article  Google Scholar 

  30. Lu, Y., et al. (2016). GeoUGV: user-generated mobile video dataset with fine granularity spatial metadata. In: Proceedings of the 7th international conference on multimedia systems, 2016. ACM.

  31. Bui, T., & Collomosse, J. (2015). Scalable sketch-based image retrieval using color gradient features. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 1–8.

  32. de Oliveira Barra, G., Lux, M., & Giro-i-Nieto, X. (2016). Large scale content-based video retrieval with LIvRE. In: 2016 14th international workshop on content-based multimedia indexing (CBMI) (pp. 1–4). IEEE.

  33. Chakraborty, S., Tickoo, O., & Iyer, R. (2015). Adaptive keyframe selection for video summarization. In: 2015 IEEE winter conference on applications of computer vision, 2015, pp. 702–709: IEEE.

  34. Schoeffmann, K., Del Fabro, M., Szkaliczki, T., Böszörmenyi, L., & Keckstein, J. (2015). Keyframe extraction in endoscopic video. Multimedia Tools and Applications, 74(24), 11187–11206.

    Article  Google Scholar 

  35. Panagiotakis, C., Papoutsakis, K., & Argyros, A. (2018). A graph-based approach for detecting common actions in motion capture data and videos. Pattern Recognition, 79, 1–11.

    Article  Google Scholar 

  36. Ejaz, N., Baik, S. W., Majeed, H., Chang, H., & Mehmood, I. (2018). Multi-scale contrast and relative motion-based key frame extraction. Journal on Image and Video Processing, 1, 40.

    Article  Google Scholar 

  37. Liu, W., Mei, T., Zhang, Y., Che, C., & Luo, J. (2015). Multi-task deep visual-semantic embedding for video thumbnail selection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3707–3715.

  38. Wu, J., Zhong, S.-H., Jiang, J., & Yang, Y. (2017). A novel clustering method for static video summarization. Multimedia Tools and Applications, 76(7), 9625–9641.

    Article  Google Scholar 

  39. Meng, J., Wang, H., Yuan, J., & Tan, Y.-P. (2016). From keyframes to key objects: Video summarization by representative object proposal selection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1039–1048.

  40. Ioannidis, A., Chasanis, V., & Likas, A. (2016). Weighted multi-view key-frame extraction. Pattern Recognition Letters., 72, 52–61.

    Article  Google Scholar 

  41. Chen, A. Y., & Corso, J. J. (2010).Propagating multi-class pixel labels throughout video frames. In 2010 Western New York image processing workshop, 2010. IEEE, pp. 14–17

  42. Liu, A.-A., Su, Y.-T., Nie, W.-Z., & Kankanhalli, M. (2017). Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1), 102–114.

    Article  Google Scholar 

  43. Wang, J., Liu, Z., Wu, Y., & Yuan, J. (2012). Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE conference on computer vision and pattern recognition, 2012. IEEE, pp. 1290–1297

  44. Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, S. W. J. I. A. (2018). Action recognition in video sequences using deep bi-directional LSTM with CNN features. IEEE Access, 6, 1155–1166.

    Article  Google Scholar 

  45. Sandeep, R., Sharma, S., Thakur, M., & Bora, P. K. (2016). Perceptual video hashing based on Tucker decomposition with application to indexing and retrieval of near-identical videos. Multimedia Tools and Applications, 75(13), 7779–7797.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Kamaraj College of Engineering and Technology, K. Vellakulam, TamilNadu, 625701, India

    T. Prathiba

  2. Department of Electronics and Communication Engineering, Mepco Schlenk Engineering College, Sivakasi, TamilNadu, 626005, India

    R. Shantha Selva Kumari

Authors
  1. T. Prathiba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Shantha Selva Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to T. Prathiba.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prathiba, T., Kumari, R.S.S. Eagle Eye CBVR Based on Unique Key Frame Extraction and Deep Belief Neural Network. Wireless Pers Commun 116, 411–441 (2021). https://doi.org/10.1007/s11277-020-07721-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-020-07721-4

Keywords

Search

Navigation