Skip to main content
SpringerLink
Log in

Facial expression recognition based on anomaly feature

  • Regular Paper
  • Published:
Optical Review Aims and scope Submit manuscript

Abstract

This study is the first attempt to recognize facial expression using face contour and facial anomaly. By extracting facial spatial–temporal–anomaly features, we constructed the anomaly features-based facial expression recognition model. The proposed model makes full use of the face structure, anomaly signal correlation, and spatial characteristics. Moreover, it reconstructs the face background required for FER while extracting the anomaly features. Then, the proposed model makes the extracted anomaly conform to facial and expression structure. The temporal anomaly and facial information are used as the representative features for FER. Finally, the extracted spatial–temporal–anomaly representative features are placed in a long short-term memory model to complete the facial expression recognition. The accuracy rate of our algorithm model exceeds 80.47% and has an advantage over related algorithms in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Sebe, N., Lew, M.S., Sun, Y., Cohen, I., Gevers, T., Huang, T.S.: Authentic facial expression analysis. Image Vis. Comput. 12, 1856–1863 (2007)

    Article  Google Scholar 

  2. Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans. Image Process. 16(1), 172–187 (2007)

    Article  ADS  MathSciNet  Google Scholar 

  3. Tang, H., Huang, T.: 3D facial expression recognition based on automatically selected features. In: Proc. IEEE Conf. Comput. Vision Pattern Recog., pp. 1–8 (2008)

  4. Xie, S., et al.: Facial expression recognition using intra-class variation reduced features and manifold regularisation dictionary pair learning. IET Comput. Vis. 12(4), 458–465 (2018)

    Article  Google Scholar 

  5. Sun, Z., et al.: Discriminative feature learning-based pixel difference representation for facial expression recognition. IET Comput. Vis. 11(8), 675–682 (2017)

    Article  Google Scholar 

  6. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  7. Savran, A., Cao, H., Nenkova, A., Verma, R.: Temporal Bayesian fusion for affect sensing: combining video, audio, and lexical modalities. IEEE Trans. Cybernet. 45(9), 1927–1941 (2015)

    Article  Google Scholar 

  8. Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)

    Article  Google Scholar 

  9. Gu, W., Xiang, C., Venkatesh, Y., Huang, D., Lin, H.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recog. 45(1), 80–91 (2012)

    Article  ADS  Google Scholar 

  10. Majumder, A., et al.: Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans. Cybern. 48(1), 103–114 (2018)

    Article  Google Scholar 

  11. Wang, Y., et al.: Facial expression recognition based on random forest and convolutional neural network. Information 10(12), 375–385 (2019)

    Article  Google Scholar 

  12. Alrubaish, H.A., et al.: The effects of facial expressions on face biometric system’s reliability. Information 11(10), 485–498 (2020)

    Article  Google Scholar 

  13. Chen, Y., et al.: Facial expression recognition via non-negative least-squares sparse coding. Information 5(2), 305–318 (2014)

    Article  Google Scholar 

  14. Fragopanagos, N., Taylor, J.G.: Emotion recognition in human–computer interaction. Neural Netw. 18(4), 389–405 (2005)

    Article  Google Scholar 

  15. Caridakis,G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A., Karpouzis, K.: Modeling naturalistic affective states via facial and vocal expressions recognition. In: Proc. Int. Conf. Multimodal Interaction, pp. 146–154 (2006)

  16. Nicolle,J., Rapp, V., Bailly, K., Prevost, L., Chetouani, M.: Robust continuous prediction of human emotions using multiscale dynamic cues. In: Proc. Int. Conf. Multimodal Interaction, pp. 501–508 (2012)

  17. Zhang, T., et al.: Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans. Cybern. 49(3), 839–847 (2019)

    Article  Google Scholar 

  18. Wollmer, M., Kaiser, M., Eyben, F., Schuller, B., Rigoll, G.: LSTM-modeling of continuous emotions in an audiovisual affect recognition framework. Image Vis. Comput. 31(2), 153–163 (2013)

    Article  Google Scholar 

  19. Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.S.: Facial expression recognition from video sequences: temporal and static modelling. Comput. Vis. Image Underst. 91, 160–187 (2003)

    Article  Google Scholar 

  20. Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 1940–1954 (2010)

    Article  Google Scholar 

  21. Pardas, M., Bonafonte, A.: Facial animation parameters extraction and expression detection using HMM. Imaging Signal Process. Image Commun. J. 17(9), 675–688 (2002)

    Article  Google Scholar 

  22. Aleksic, P.S., Katsaggelos, A.K.: Automatic facial expression recognition using facial animation parameters and multistream HMMs. IEEE Trans. Inf. Forens. Secur. 1(1), 3–11 (2006)

    Article  Google Scholar 

  23. Wu,C., Wang, S., Ji, Q.: Multi-instance hidden Markov model for facial expression recognition. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–6 (2015)

  24. Walecki,R., Rudovic, O., Pavlovic, V., Pantic, M.: Variable state latent conditional random fields for facial expression recognition and action unit detection. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–8 (2015)

  25. Ranzato,M., Susskind, J., Mnih, V., Hinton, G.: On deep generative models with applications to recognition. In: Proc. IEEE Conf. Comput. Vision Pattern Recog., pp. 2857–2864 (2011)

  26. Liu,M., Shan, S., Wang, R., Chen, X.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proc. IEEE Conf. Comput. Vision Pattern Recog., pp. 1749–1756 (2014)

  27. Song,I., Kim, H.-J., Jeon, P. B.: Deep learning for real-time robust facial expression recognition on a smartphone. In: Proc. IEEE Int. Conf. Consumer Electron., pp. 564–567 (2014)

  28. Liu,M., Li, S., Shan, S., Chen, X.: AU-aware deep networks for facial expression recognition. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–6 (2013)

  29. Meng, H., et al.: Time-delay neural network for continuous emotional dimension prediction from facial expression sequences. IEEE Trans. Cybern. 46(4), 916–929 (2016)

    Article  Google Scholar 

  30. Liu,P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In: Proc. IEEE Conf. Comput. Vision Pattern Recog., pp. 1805–1812 (2014)

  31. Berretti, S., Amor, B.B., Daoudi, M., Del Bimbo, A.: “3D facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. Int. J. Comput. Graph. 27(11), 1021–1036 (2011)

    Google Scholar 

  32. Lemaire,P., Ardabilian, M., Chen, L., Daoudi, M.: Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–7 (2013)

  33. Dapogny,A., Bailly, K., Dubuisson, S.: Dynamic facial expression recognition by joint static and multi-time gap transition classification. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–6 (2015)

  34. Kahou, S.E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al.: Emonets: Multimodal deep learning approaches for emotion recognition in video. J. Multimod. User Interfaces 10(2), 99–111 (2016)

    Article  Google Scholar 

  35. Liu,M., Wang, R., Li, S., Shan, S., Huang, Z., Chen, X.: Combining multiple kernel methods on Riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th International Conference on Multimodal Interaction. ACM, pp. 494–501 (2014)

  36. Ding,W., Xu, M., Huang, D., Lin, W., Dong, M., Yu, X., Li, H.: Audio and face video emotion recognition in the wild using deep neural networks and small datasets. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, pp. 506–513 (2016)

  37. Xu,B., Fu, Y., Jiang, Y.-G., Li, B., Sigal, L.: Video emotion recognition with transferred deep feature encodings. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. ACM, pp. 15–22 (2016)

  38. Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gulcehre, C., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., et al.: Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction. ACM, pp. 543–550 (2013)

  39. Bargal,S.A., Barsoum, E., Ferrer, C.C., Zhang, C.: Emotion recognition in the wild from videos using images. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, pp. 433–436 (2016)

  40. Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N., Yan, S.: Peak-piloted deep network for facial expression recognition. In: European conference on computer vision. Springer, pp. 425–442 (2016)

  41. Yu, Z., Liu, Q., Liu, G.: Deeper cascaded peak-piloted network for weak expression recognition. Vis. Comput. 34, 1–9 (2017)

    ADS  Google Scholar 

  42. Kim, D.H., Baddar, W., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 10(2), 223–236 (2019)

    Article  Google Scholar 

  43. Kim,Y., Yoo, B., Kwak, Y., Choi, C., Kim, J.: Deep generative contrastive networks for facial expression recognition (2017). arXiv preprint arXiv: 1703.07140

  44. Chen, J., Xu, R., Liu, L.: Deep peak-neutral difference feature for facial expression recognition. Multimedia Tools Appl. 77, 1–17 (2018)

    Article  Google Scholar 

  45. Le,Q.V., Jaitly, N., Hinton, G.E.: A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 1504.00941 (2015)

  46. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  ADS  Google Scholar 

  47. Al Chanti, D.A., Caplier, A.: Deep learning for spatio-temporal modeling of dynamic spontaneous emotions. In: IEEE Transactions on Affective Computing, pp. 1–1 (2018)

  48. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  49. Abbasnejad, I., Sridharan, S., Nguyen, D., Denman, S., Fookes, C., Lucey, S.: Using synthetic data to improve facial expression analysis with 3d convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1609–1618 (2014)

  50. Barros, P., Wermter, S.: Developing crossmodal expression recognition based on a deep neural model. Adapt. Behav. 24(5), 373–396 (2016)

    Article  Google Scholar 

  51. Zhao, J., Mao, X., Zhang, J.: Learning deep facial expression features from image and optical flow sequences using 3d cnn. Vis. Comput. 34, 1–15 (2018)

    Article  ADS  Google Scholar 

  52. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)

    Article  Google Scholar 

  53. Pini,S., Ahmed, O.B., Cornia, M., Baraldi, L., Cucchiara, R., Huet, B.: Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, pp. 536–543 (2017)

  54. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: 2015 IEEE International Conference in Computer Vision on (ICCV). IEEE, pp. 2983–2991 (2015)

  55. Mollahosseini,A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: Proc. IEEE Winter Conf. Appl. Comput. Vis., pp. 1–10 (2016)

  56. Huang,M.-W., Wang, Z.-W., Ying, Z.-L.: A new method for facial expression recognition based on sparse representation plus LBP. In: Proc. Int. Congr. Image Signal Process., pp. 1750–1754 (2010)

  57. Zhen,W., Zilu, Y.: Facial expression recognition based on local phase quantization and sparse representation. In: Proc. IEEE Int. Conf. Natural Comput., pp. 222–225 (2012)

  58. Kumawat,S., Verma, M., Raman, S. Lbvcnn: Local binary volume convolutional neural network for facial expression recognition from image sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

  59. Jung,H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proc. IEEE Int. Conf. Comput. Vis., pp. 2983–2991 (2015)

  60. Lee, S.H., Baddar, W.J., Ro, Y.M.: Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recogn. 54, 52–67 (2016)

    Article  ADS  Google Scholar 

  61. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 650–660 (1990)

    Article  Google Scholar 

  62. Ringach, D.L.: Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Trans. Image Process. 5(1), 1582–1586 (1996)

    ADS  Google Scholar 

  63. Rafael, C.G., et al.: Digital Image Processing Using Matlab, p. 366. Publishing House of Electronics Industry, Beijing (2013)

    Google Scholar 

  64. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)

    Article  Google Scholar 

  65. Markus, M., Breunig, et al.: LOF: identifying density-based local outliers. In: SIGMOD '00: Proceedings of the 2000 ACM SIGMOD International Conference on Management of dataMay, pp. 93–104 (2000)

  66. Muller, M., Baier, G., Rummel, C., Schindler, K., Stephani, U.: A multivariate approach to correlation analysis based on random matrix theory. In: Schelter, B., Timmer, J., Bonhage, S. (eds.) Seizure prediction in epilepsy: from basic mechanisms to clinical applications, pp. 209–226. Wiley, New York (2008)

    Chapter  Google Scholar 

  67. Bach, F., Jordan, M.: Learning spectral clustering. In: Thrun, S., Saul, L., Schoellkopf, S. (eds.) Advances in neural information processing systems 16 (NIPS), p. 305312. MIT, Cambridge (2004)

    Google Scholar 

  68. Maesschalck, R., et al.: The Mahalanobis distance. Chemom. Intell. Lab. Syst. 50(1), 1–18 (2000)

    Article  Google Scholar 

  69. Valstar,M., Pantic, M.: Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, p. 65 (2010)

  70. Yan,W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., Fu, X.: CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: Proc. IEEE Int. Conf. Autom. Face Gesture Recog., pp. 1–7 (2013)

  71. Yan, W.J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), 604–615 (2014)

    Article  Google Scholar 

  72. Sun,N., Li, Q., Huan, R., Liu, J., Han, G.: Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn. Lett. (2017)

  73. Donahue,J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634 (2015)

  74. Simonyan,K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568–576 (2014)

  75. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  76. Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H.: Efficient spatiotemporal local binary patterns for spontaneous facial micro-expression recognition. PLoS ONE 10, e0124674 (2015)

    Article  Google Scholar 

  77. Oh,Y.-H., Le Ngo, A. C., See, J., Liong, S.-T., Phan, R. C.-W., Ling, H.-C.: Monogenic Riesz wavelet representation for microexpression recognition. In: Proc. IEEE Int. Conf. Digital Signal Process., pp. 1237–1241 (2015)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant no. 61866015).

Author information

Authors and Affiliations

  1. School of Software and Internet of Things Engineering, Jiangxi University of Finance and Economics, Nanchang, China

    Kan Hong

Authors
  1. Kan Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kan Hong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, K. Facial expression recognition based on anomaly feature. Opt Rev 29, 178–187 (2022). https://doi.org/10.1007/s10043-022-00734-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10043-022-00734-3

Keywords

Search

Navigation