Skip to main content
SpringerLink
Log in

Multi-level feature fusion model-based real-time person re-identification for forensics

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Person forensics aims to retrieve the specified person across non-overlapping cameras. It is difficult owing to the appearance variations caused by occlusion, human pose change, background clutter, illumination variation, etc. In this scenario, current models face great challenges in extracting effective features. Recent deep learning models mainly focus on extracting representative deep features to cope with appearance variations, while handcrafted features are not fully explored. In this paper, a multi-level feature fusion model (MFFM) is designed to combine both deep features and handcrafted features in real time. MFFM is first utilized to describe person appearance. Then, local binary pattern (LBP) and histogram of oriented gradient (HOG) are extracted to cope with geometric change and illumination variance. Using LBP and HOG, 11.89% on the CUHK03, 15.30% on the Market-1501 and 8.25% on the VIPeR top-1 recognition accuracy improvement for the proposed method are achieved with only 9.66%, 4.90%, and 7.59% extra processing time. Experimental results indicate MFFM can achieve the best performance compared to the state-of-the-art models on the Market1501, CUHK03, and VIPeR datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Zhou, Z., Wu, J.Q.M Sun, X.: Multiple distance-based coding: toward scalable feature matching for large-scale web image search. IEEE Transactions on Big Data, pp 1–1 (2019). https://doi.org/10.1109/TBDATA.2019.2919570

  2. Qi, L., Wang, R., Hu, C., et al.: Time-aware distributed service recommendation with privacy-preservation. Inf. Sci. 480, 354–364 (2019)

    Article  Google Scholar 

  3. Qi, L., Zhang, X., Dou, W., Hu, C., Yang, C., Chen, J.: A two-stage locality-sensitive hashing based approach for privacy-preserving mobile service recommendation in cross-platform edge environment. Future Gener Computer Syst 88, 636–643 (2018)

    Article  Google Scholar 

  4. Qi, L., Chen, Y., Yuan, Y., et al. A QoS-aware virtual machine scheduling method for energy conservation in cloud-based cyber-physical systems. World Wide Web (2019). https://doi.org/10.1007/s11280-019-00684-y

    Article  Google Scholar 

  5. Qu, Z., Zhu, T., Wang, J., Wang, X.: A novel quantum steganography based on brown states. Computers, Mater Continua. 56, 47–59 (2018)

    Google Scholar 

  6. Cui, Q., McIntosh, S., Sun, H.: Identifying materials of photographic images and photorealistic computer generated graphics based on deep CNNs. Computers, Mater Continua. 55, 229–243 (2018)

    Google Scholar 

  7. Meng, R., Rice, S., Wang, J., Sun, X.: A fusion steganographic algorithm based on faster R-CNN. Computers, Mater Continua. 55, 001–016 (2018)

    Google Scholar 

  8. McLaughlin, N., Rincon, J.M., Miller, P.C.: Person reidentification using deep ConvNets with multitask learning. IEEE Trans. Circuits Syst. Video Technol. 27, 525–539 (2017)

    Article  Google Scholar 

  9. Liu, H., Feng, J., Qi, M., Jiang, J., Yan, S.: End-to-end comparative attention networks for person re-identification. IEEE Trans. Image Process. 26, 1–2 (2016)

    MathSciNet  MATH  Google Scholar 

  10. Li, W., Zhu, X., Gong, S.: Harmonious attention network for person re-identification. In: CVPR, pp. 285–2294 (2018)

  11. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 426–444 (2015)

    Article  Google Scholar 

  12. Hariharan, B., Arbelaez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 447–456 (2015). https://doi.org/10.1109/cvpr.2015.7298642

  13. Jin, X., Chen, Y., Dong, J., Feng, J., Yan, S..: Collaborative layer-wise discriminative learning in deep neural networks. In ECCV, pp. 71–749 (2016)

    Chapter  Google Scholar 

  14. Schumann, A. Stiefelhagen, R.: Person re-identification by deep learning attribute-complementary information. In CVPR Workshops, pp. 1437–1443. IEEE, Hawaii (2017)

  15. Lin, Y., Zheng, L., Zheng, Z., Wu, Y., Yang, Y.: Improving person re-identification by attribute and identity learning (2017). arXiv:1703.07220

  16. Khamis, S., Kuo, C.H., Singh, V.K., Shet, V.D., Davis, L.S.: Joint learning for attribute-consistent person re-identification. In: Agapito, L., Bronstein, M., Rother, C. (eds,) Computer Vision - ECCV 2014 Workshops. ECCV 2014. Lecture Notes in Computer Science, vol 8927, pp. 134-146. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16199-0_10

    Chapter  Google Scholar 

  17. Matsukawa, T., Suzuki, E.: Person re-identification using cnn features learned from combination of attributes. In: ICPR, pp. 2448–2435 (2016)

  18. Wang, J., Zhu, X., Gong, S., Li, W.: Transferable joint attribute-identity deep learning for unsupervised person re-identification. In: ICCV, pp. 2275–2284 (2018)

  19. Liu, X., Zhao, H., Tian, M., Sheng, L., Shao, J., Yi, S., Yan, J., Wang, X.: Hydraplus-net: Attentive deep features for pedestrian analysis. In: ICCV, pp. 370–379 (2017)

  20. Zhao, H., Tian, M., Sun, S., Shao, J., Yan. J., Yi, S., Wang, X., Tang, X.: Spindle net: Person re-identification with human body region guided feature decomposition and fusion. In: CVPR, pp. 1077–1085 (2017)

  21. Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q.: Scalable person re-identification: A benchmark. In: ICCV, pp. 1116–1124 (2015)

  22. Li, W., Zhao, R., Xiao, T., Wang, X.: Deep reid: Deep filter pairing neural network for person re-identification. In: CVPR, pp. 1116–1124 (2014)

  23. Zheng, Z., Zheng, L., Yang, Y.: Unlabeled samples generated by Gan improve the person re-identification baseline in vitro. In: ICCV, pp. 3954–3962 (2017)

  24. Chen, D., Yuan, Z., Chen, B., Zheng, N.: Similarity learning with spatial constraints for person re-identification. In CVPR, pp. 1268–1277. IEEE Computer Society, Las Vegas (2016)

  25. Chen, D., Yuan, Z., Hua, G., Zheng, N., Wang, J.: Similarity learning on an explicit polynomial kernel feature map for person re-identification. In: CVPR, pp. 1565–1573 (2015)

  26. Li, W., Wang, X.: Locally aligned feature transforms across views. In: CVPR, pp. 3794–3801 (2013)

  27. Liao, S., Hu, Y., Zhu, X., Li, S. Z..: Person re-identification by local maximal occurrence representation and metric learning. In: CVPR, pp. 2197–2206. IEEE Computer Society, Boston (2015)

  28. Matsukawa, T., Okabe, T., Suzuki, E., Sato, Y.: Hierarchical gaussian descriptor for person re-identification. In: CVPR, pp. 1383–1392 (2016)

  29. Shi, Z., Hospedales, T. M., Xiang, T.: Transferring a semantic representation for person re-identification and search. In: CVPR, pp. 4404–4413 (2015)

  30. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR, pp. 1–9 (2015)

  31. Cheng, D., Gong, Y., Zhou, S., Wang, J., Zheng, N..: Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In: CVPR, pp. 1337–1364 (2016)

  32. Su, C., Zhang, S., Xing, J., Gao, W., Tian, Q.: (2016) Deep attributes driven multi-camera person re-identification. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9906, pp. 475–491. Springer, Cham. https://doi.org/10.1007/978-3-319-46475-6_30

    Chapter  Google Scholar 

  33. Ustinova, E., Ganin, Y., Lempitsky, V.: Multiregion bilinear convolutional neural networks for person reidentification. In: IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 1–6 (2017)

  34. Varior, R., Haloi, M., Wang, G.: Gated Siamese convolutional neural network architecture for human reidentification. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV, LNCS, vol. 9912, pp. 791–808. Springer, Cham (2016)

    Chapter  Google Scholar 

  35. Wu, L., Shen, C., Hengel, A. v. d.: Personnet: Person re-identification with deep convolutional neural networks (2016). arXiv preprint arXiv:1601.07255

  36. Zhao, R., Ouyang, W., Wang, X.: Unsupervised salience learning for person re-identification. In: CVPR, pp. 3586–3593 (2013). https://doi.org/10.1109/CVPR.2013.460

  37. Zhao, R., Ouyang, W., Wang, X.: Person re-identification by salience matching. In: ICCV, pp. 2528–2535 (2013)

  38. Zhao, R., Ouyang, W., Wang, X.: Learning mid-level filters for person re-identification. In: CVPR, pp. 144–151. IEEE (2014)

  39. Das, A., Chakraborty, A., Roy-Chowdhury, A.K. (2014) Consistent re-identification in a camera network. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8690, pp. 330–345. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_22

    Chapter  Google Scholar 

  40. Li, Z., Chang, S., Liang, F., Huang, T., Cao, L., Smith, J.: Learning locally-adaptive decision functions for person verification. In: CVPR, pp. 3610–3617, IEEE (2013)

  41. Pedagadi, S., Orwell, J., Velastin, S., Boghossian, B.: Local fisher discriminant analysis for pedestrian re-identification. In: CVPR, pp. 3318–3325 (2013)

  42. Liu, X., Song, M., Tao, D., Zhou, X. Chen, C., Bu, J.: Semi-supervised coupled dictionary learning for person reidentification. In CVPR, pp. 3550–3557 (2014)

  43. Yang, Y., Yang, J., Yan, J., Liao, S., Yi, D., Li, S.: Salient color names for person re-identification. In: ECCV, pp. 536–551 (2014)

  44. Liao, S., Hu, Y., Zhu, X., Li, S.: Person re-identification by local maximal occurrence representation and metric learning. In: CVPR, pp. 2197–2206 (2015)

  45. Van De Weijer, J., Schmid, C., Verbeek, J., Larlus, D.: Learning color names for real-world applications. Trans. Image. Process. 18(7), 1512–1523 (2009)

    Article  MathSciNet  Google Scholar 

  46. Matsukawa, T., Okabe, T., Suzuki, E., Sato, Y.: Hierarchical gaussian descriptor for person re-identification. In: CVPR, pp. 1363–1372 (2016)

  47. Ma, B., Su, Y., Jurie, F.: Bicov: a novel image representation for person reidentification and face verification. In: BMVC (2012)

  48. Farenzena, M., Bazzani, L., Perina, A., Murino, V., Cristani, M.: Person reidentification by symmetry-driven accumulation of local features. In: CVPR, pp. 2360–2367 (2010)

  49. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 311–341 (2015)

  50. Fan, H., Mei, X., Prokhorov, D., Ling, H.: Multi-level contextual RNNs with attention model for scene labeling. IEEE Trans. Intell. Transp. Syst. 19, 3475–3485 (2016)

    Article  Google Scholar 

  51. Yang, S., Ramanan, D.: Multi-scale recognition with dagcnns. In: ICCV, pp. 1215–1223 (2015)

  52. Cai, S., Zuo, W., Zhang, L.: Higher-order integration of hierarchical convolutional activations for fine-grained visual categorization. In: CVPR, pp. 511–520 (2017)

  53. Yu, W., Yang, K., Yao, H., Sun, X., Xu, P.: Exploiting the complementary strengths of multi-layer cnn features for image retrieval. Neurocomputing 239, 235–240 (2017)

    Article  Google Scholar 

  54. Yang, X., Molchanov, P., Kautz, J..: Multilayer and multimodal fusion of deep neural networks for video classification. In: ACM MM, pp. 978–987 (2016)

  55. Gray, D., Brennan, S., Tao, H.: Evaluating appearance models for recognition, reacquisition, and tracking. In: IEEE International Workshop on Performance Evaluation for Tracking and Surveillance (PETS), pp. 1–7 (2007)

  56. Liao, S., Li, S. Z.: Efficient PSD constrained asymmetric metric learning for person re-identification. In: ICCV, pp. 3885–3893 (2015)

  57. Zhang, L., Xiang, T., Gong, S.: Learning a discriminative null space for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '16), pp. 1239–1248. Las Vegas, NV, USA (2016)

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of China (U1803262, 61602349, and 61440016).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430065, China

    Shiqin Wang, Xin Xu & Lei Liu

  2. Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430065, China

    Xin Xu

  3. Artificial Intelligence Institute, Wuhan University of Science and Technology, Wuhan, 430065, China

    Shiqin Wang, Xin Xu & Lei Liu

  4. Institute of Systems Science, National University of Singapore, Singapore, 119615, Republic of Singapore

    Jing Tian

Authors
  1. Shiqin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Xu, X., Liu, L. et al. Multi-level feature fusion model-based real-time person re-identification for forensics. J Real-Time Image Proc 17, 73–81 (2020). https://doi.org/10.1007/s11554-019-00908-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-019-00908-4

Keywords

Search

Navigation