Skip to main content

Advertisement

SpringerLink
Log in

Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Since the distinction between two expressions is fairly vague, usually a subtle change in one part of the human face is enough to change a facial expression. Most of the existing facial expression recognition algorithms are not robust enough because they rely on general facial features or algorithms without considering differences between facial expression and facial identity. In this paper, we propose a person-independent recognition method based on Wasserstein generative adversarial networks for micro-facial expressions, where a facial expression recognition network and a facial identity recognition network are established to improve the accuracy and robustness of facial expression recognition via inhibition of intra-class variation. Extensive experimental results demonstrate that 90% average recognition accuracy of facial expression has been reached on a mixed dataset composed of CK+, Multi-PIE, and JAFFE. Moreover, our method achieves 96% accuracy of person-independent recognition on CK+. A 4.5% performance gain is achieved with the novel identity-inhibited expression feature. The proposed algorithm in this paper has been successfully applied to Haikang Visual Integrated Management Platform (iVMS-8700). At present, it runs well and can effectively recognize facial expressions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Xia, K., Yin, H., Zhang, Y.: Deep semantic segmentation of kidney and space-occupying lesion area based on SCNN and ResNet models combined with SIFT-flow algorithm. J. Med. Syst. 87(43), 1253–1268 (2019)

    Google Scholar 

  2. Shan, C., Gong, S., Mcowan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  3. Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)

    Article  Google Scholar 

  4. Cohen, I., Sebe, N., Garg, A., et al.: Facial expression recognition from video sequences. In: Proceedings. IEEE International Conference on Multimedia and Expo, vol. 24, no. 2, pp. 121–124 (2002)

  5. Lopes, A.T., Aguiar, E.D., Souza, A.F.D., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2016)

    Article  Google Scholar 

  6. Lee, S.H., Plataniotis, K.N.K., Ro, Y.M.: Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans. Affect. Comput. 5(3), 340–351 (2014)

    Article  Google Scholar 

  7. Liu, M., Shan, S., Wang, R., et al.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. IEEE Trans. Image Process. 25(12), 5920–5932 (2016)

    Article  MathSciNet  Google Scholar 

  8. Yanpeng, L., Yibin, L., Xin, M., et al.: Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17(4), 712–729 (2017)

    Article  Google Scholar 

  9. Jiang, Y., Chung, F.-L., Ishibuchi, H., et al.: Multitask TSK fuzzy system modeling by mining intertask common hidden structure. IEEE Trans. Cybern. 45(3), 548–561 (2015)

    Article  Google Scholar 

  10. Li, H., Sun, J., Xu, Z., et al.: Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans. Multimed. 2017, 1 (2017)

    Google Scholar 

  11. Ghimire, D., Jeong, S., Joonwhoan, L.: Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl. 76(6), 7803–7821 (2017)

    Article  Google Scholar 

  12. Jiang, Y., Chung, F.-L., Wang, S., Deng, Z., Wang, J., Qian, P.: Collaborative fuzzy clustering from multiple weighted views. IEEE Trans. Cybern. 45(4), 688–701 (2015)

    Article  Google Scholar 

  13. Valstar, M.F., Sánchez-Lozano, E., Cohn, J.F., et al.: Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, 30 May–3 June 2017, pp. 839–847 (2017)

  14. Bazrafkan, S., Nedelcu, T., Filipczuk, P., et al.: Deep learning for facial expression recognition: a step closer to a smartphone that knows your moods. In: 2017 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 8–10 January 2017, pp. 1208–1223 (2017)

  15. Hasani, B., Arzani, M.M., Fathy, M., et al.: Facial expression recognition with discriminatory graphical models. In: 2016 2nd International Conference of Signal Processing and Intelligent Systems (ICSPIS), Tehran, Iran, 14–15 December 2016, pp. 1–7 (2016)

  16. Meng, Z., Liu, P., Cai, J., et al.: Identity-aware convolutional neural network for facial expression recognition. IEEE Comput. Soc. 24(23), 562–573 (2017)

    Google Scholar 

  17. Hsu, S.C., Huang, H.H., Huang, C.L.: Facial expression recognition for human-robot interaction. In: 2017 First IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan, 10–12 April 2017. IEEE (2017)

  18. Zhang, C., Wang, P., Chen, K., et al.: Identity-aware convolutional neural networks for facial expression recognition. J. Syst. Eng. Electron. 28(4), 784–792 (2017)

    Article  Google Scholar 

  19. Uddin, M.Z., Hassan, M.M., Almogren, A., et al.: A facial expression recognition system using robust face features from depth videos and deep learning. Comput. Electr. Eng. 63, 114–125 (2017)

    Article  Google Scholar 

  20. Qian, P., Jiang, Y., Deng, Z., Hu, L., Sun, S., Wang, S., Muzic R.F. Jr.: Cluster prototypes and fuzzy memberships jointly leveraged cross-domain maximum entropy clustering. IEEE Trans. Cybern. 46(1), 181–193 (2016)

    Article  Google Scholar 

  21. Liu, X., Kumar, B.V.K.V, You, J., et al.: Adaptive deep metric learning for identity-aware facial expression recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017, pp. 20–29 (2017)

  22. Baddar, W.J., Kim, D.H., Ro, Y.M.: Learning features robust to image variations with siamese networks for facial expression recognition. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) International Conference on Multimedia Modeling, pp.189–200. Springer, Cham (2017)

    Google Scholar 

  23. Li, J., Wang, S., Huo, J., Guo, H., Zhao, H., Mei, Y.: Bayesian framework with non-local and low-rank constraint for image reconstruction. J. Phys. Conf. Ser. 787, 012008 (2017)

    Article  Google Scholar 

  24. Sariyanidi, E., Gunes, H., Cavallaro, A.: Learning bases of activity for facial expression recognition. IEEE Trans. Image Process. 26(4), 1965–1978 (2017)

    Article  MathSciNet  Google Scholar 

  25. Jang, J., Kim, D.H., Kim, H.I., et al.: Color channel-wise recurrent learning for facial expression recognition. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017, pp. 1233–1237 (2017)

  26. Xia, K.J., Yin, H.S., Wang, J.Q.: A novel improved deep convolutional neural network model for medical image fusion. Cluster Comput. 24(3), 1–13 (2018)

    Google Scholar 

  27. Fathallah, A., Abdi, L., Douik, A.: Facial expression recognition via deep learning. In: 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), Hammamet, Tunisia, 30 October–3 November 2017, vol. 127, no. 99, pp. 107–114. IEEE Computer Society (2017)

  28. Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 22–29 October 2017, pp. 2783–2788 (2017)

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering, University of Yamanashi, Kofu, 4000016, Japan

    Caie Xu

  2. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310000, Zhejiang, China

    Yang Cui, Yunhui Zhang, Peng Gao & Jiayi Xu

Authors
  1. Caie Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yunhui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiayi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Caie Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, C., Cui, Y., Zhang, Y. et al. Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware. Multimedia Systems 26, 53–61 (2020). https://doi.org/10.1007/s00530-019-00628-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-019-00628-6

Keywords

Search

Navigation