Skip to main content
SpringerLink
Log in

4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

One of the main challenges in dynamic facial expression recognition is how to capture temporal information in the system. In this study, a novel approach based on time series analysis is adapted for this problem. The proposed dynamic facial expression recognition system comprises four phases: head pose correction and normalization, feature extraction, feature selection and classification. Head pose detection and correction is the first phase to realign locations of the facial landmarks. A comprehensive set of geometric deformations including point, distance and angle deformations are extracted from the key points. The concept of facial action unit analysis is interlocked with this phase to identify related key points from the landmarks. A set of multimodal time series are then constructed from the extracted deformations by applying a sliding window to characterize the dynamics of mean deformations in a window. In the third phase, a feature selection method based on neighborhood component analysis is applied on the peak value of deformation features to select useful features and discard irrelevant ones. Finally, adaptive cost dynamic time warping is utilized to recognize six prototypic expressions from multimodal time series of selected features. Experimental results on BU-4DFE data set confirm that proposed algorithm is efficient in dynamic facial expression recognition compared with state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Amor, B.B., Drira, H., Berretti, S., Daoudi, M., Srivastava, A.: 4-D facial expression recognition by learning geometric deformations. IEEE Trans. Cybern. 44(12), 2443–2457 (2014)

    Article  Google Scholar 

  2. An, F., Liu, Z.: Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Vis. Comput. (2019). https://doi.org/10.1007/s00371-019-01635-4

    Article  Google Scholar 

  3. Arcoverde Neto, E.N., Duarte, R.M., Barreto, R.M., Magalhães, J.P., Bastos, C.C.M., Ren, T.I., Cavalcanti, G.D.C.: Enhanced real-time head pose estimation system for mobile device. Integr. Comput.-Aided Eng. 21(3), 281–293 (2014)

    Article  Google Scholar 

  4. Berndt, D.J., Clifford, J.: Finding patterns in time series: a dynamic programming approach. In: Advances in Knowledge Discovery and Data Mining, pp. 229–248 (1996)

  5. Berretti, S., del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis. Comput. 29(12), 1333–1350 (2013)

    Article  Google Scholar 

  6. Bolourchi, P., Demirel, H., Uysal, S.: Target recognition in SAR images using radial Chebyshev moments. Signal Image Video Process. 11(6), 1033–1040 (2017)

    Article  Google Scholar 

  7. Carcagnì, P., Del Coco, M., Leo, M., Distante, C.: Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1), 645 (2015)

    Article  Google Scholar 

  8. Chen, J., Chen, Z., Chi, Z., Fu, H.: Dynamic texture and geometry features for facial expression recognition in video. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4967–4971. IEEE (2015)

  9. Choi, J.Y.: Spatial pyramid face feature representation and weighted dissimilarity matching for improved face recognition. Vis. Comput. 34(11), 1535–1549 (2018)

    Article  Google Scholar 

  10. Derkach, D., Ruiz, A., Sukno, F.M.: Head pose estimation based on 3-D facial landmarks localization and regression. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), pp. 820–827. IEEE (2017)

  11. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  12. Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)

    Article  Google Scholar 

  13. Fang, T., Zhao, X., Shah, S.K., Kakadiaris, I.A.: 4D facial expression recognition. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1594–1601 (2011)

  14. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)

    Article  Google Scholar 

  15. Ghimire, D., Lee, J., Li, Z.N., Jeong, S.: Recognition of facial expressions based on salient geometric features and support vector machines. Multimed. Tools Appl. 76(6), 7921–7946 (2017)

    Article  Google Scholar 

  16. Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. (2018). https://doi.org/10.1007/s00371-018-1585-8

    Article  Google Scholar 

  17. Goh, K.M., Ng, C.H., Lim, L.L., Sheikh, U.U.: Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. (2018). https://doi.org/10.1007/s00371-018-1607-6

    Article  Google Scholar 

  18. Goldberger, J., Hinton, G.E., Roweis, S.T., Salakhutdinov, R.R.: Neighbourhood components analysis. In: Saul, L.K., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 17, pp. 513–520. MIT Press, Cambridge (2005)

    Google Scholar 

  19. Górecki, T., Łuczak, M.: Multivariate time series classification with parametric derivative dynamic time warping. Expert Syst. Appl. 42(5), 2305–2312 (2015)

    Article  Google Scholar 

  20. Guo, Y., Zhao, G., Pietikainen, M.: Dynamic facial expression recognition with atlas construction and sparse representation. IEEE Trans. Image Process. 25(5), 1977–1992 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  21. Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)

    Article  Google Scholar 

  22. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2983–2991 (2015)

  23. Kalsum, T., Anwar, S.M., Majid, M., Khan, B., Ali, S.M.: Emotion recognition from facial expressions using hybrid feature descriptors. IET Image Process. 12(6), 1004–1012 (2018)

    Article  Google Scholar 

  24. Keogh, E., Kasetty, S.: On the need for time series data mining benchmarks: a survey and empirical demonstration. Data Min. Knowl. Discov. 7(4), 349–371 (2003)

    Article  MathSciNet  Google Scholar 

  25. Kołakowska, A., Landowska, A., Szwoch, M., Szwoch, W., Wróbel, M.R.: Emotion Recognition and Its Applications, pp. 51–62. Springer, Berlin (2014)

    Google Scholar 

  26. Li, K., Jin, Y., Akram, M.W., Han, R., Chen, J.: Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis. Comput. (2019). https://doi.org/10.1007/s00371-019-01627-4

    Article  Google Scholar 

  27. Li, W., Huang, D., Li, H., Wang, Y.: Automatic 4D facial expression recognition using dynamic geometrical image network. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 24–30. IEEE (2018)

  28. Liang, D., Liang, H., Yu, Z., Zhang, Y.: Deep convolutional BiLSTM fusion network for facial expression recognition. Vis. Comput. (2019). https://doi.org/10.1007/s00371-019-01636-3

    Article  Google Scholar 

  29. Lien, J.J.J., Kanade, T., Cohn, J.F., Li, C.C.: Detection, tracking, and classification of action units in facial expression. Robot. Auton. Syst. 31(3), 131–146 (2000)

    Article  Google Scholar 

  30. Lopes, A.T., de Aguiar, E., Souza, A.F.D., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)

    Article  Google Scholar 

  31. Reale, M., Zhang, X., Yin, L.: Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)

  32. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: 2011 IEEE International Conference on Face and Gesture, pp. 406–413. IEEE (2011)

  33. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Image Vis. Comput. 30(10), 762–773 (2012)

    Article  Google Scholar 

  34. Shao, J., Gori, I., Wan, S., Aggarwal, J.: 3D dynamic facial expression recognition using low-resolution videos. Pattern Recognit. Lett. 65, 157–162 (2015)

    Article  Google Scholar 

  35. Sun, Y., Chen, X., Rosato, M., Yin, L.: Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 40(3), 461–474 (2010)

    Article  Google Scholar 

  36. Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)

    Article  Google Scholar 

  37. Valstar, M.F., Pantic, M.: Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(1), 28–43 (2012)

    Article  Google Scholar 

  38. Wan, Y., Chen, X.L., Shi, Y.: Adaptive cost dynamic time warping distance in time series analysis for classification. J. Comput. Appl. Math. 319, 514–520 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  39. Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., Kissler, J.: Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE 12(5), e0177,239 (2017)

    Article  Google Scholar 

  40. Xue, M., Mian, A., Liu, W., Li, L.: Automatic 4D facial expression recognition using DCT features. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 199–206. IEEE (2015)

  41. Yang, W., Wang, K., Zuo, W.: Neighborhood component feature selection for high-dimensional data. J. Comput. 7(1), 31–37 (2012)

    Google Scholar 

  42. Yao, Y., Huang, D., Yang, X., Wang, Y., Chen, L.: Texture and geometry scattering representation-based facial expression recognition in 2D+3D videos. ACM Trans. Multimed. Comput. Commun. Appl. 14(1s), 18:1–18:23 (2018)

    Article  Google Scholar 

  43. Yeganli, S.F., Demirel, H., Yu, R.: Noise removal from mr images via iterative regularization based on higher-order singular value decomposition. Signal Image Video Process. 11(8), 1477–1484 (2017)

    Article  Google Scholar 

  44. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, pp. 1–6. IEEE (2008)

  45. Yurtkan, K., Demirel, H.: Feature selection for improved 3D facial expression recognition. Pattern Recognit. Lett. 38, 26–33 (2014)

    Article  Google Scholar 

  46. Zarbakhsh, P., Demirel, H.: Low-rank sparse coding and region of interest pooling for dynamic 3D facial expression recognition. Signal Image Video Process. 12(8), 1611–1618 (2018)

    Article  Google Scholar 

  47. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans. Multimed. 18(12), 2528–2536 (2016)

    Article  Google Scholar 

  48. Zhao, J., Mao, X., Zhang, J.: Learning deep facial expression features from image and optical flow sequences using 3D CNN. Vis. Comput. 34(10), 1461–1475 (2018)

    Article  Google Scholar 

  49. Zhao, J., Wang, S.H., Liu, X., Liu, Y., Chen, Y.Q.: Early diagnosis of cirrhosis via automatic location and geometric description of liver capsule. Vis. Comput. 34(12), 1677–1689 (2018)

    Article  Google Scholar 

  50. Zhen, Q., Huang, D., Drira, H., Amor, B.B., Wang, Y., Daoudi, M.: Magnifying subtle facial motions for effective 4D expression recognition. IEEE Trans. Affect. Comput. (2017). https://doi.org/10.1109/TAFFC.2017.2747553

    Article  Google Scholar 

  51. Zhen, Q., Huang, D., Wang, Y., Chen, L.: Muscular movement model-based automatic 3D/4D facial expression recognition. IEEE Trans. Multimed. 18(7), 1438–1450 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical and Electronic Engineering Department, Eastern Mediterranean University, via Mersin 10, Famagusta, Northern Cyprus, Turkey

    Payam Zarbakhsh & Hasan Demirel

  2. Faculty of Engineering, Cyprus International University, via Mersin 10, Nicosia, Northern Cyprus, Turkey

    Payam Zarbakhsh

Authors
  1. Payam Zarbakhsh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hasan Demirel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Payam Zarbakhsh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zarbakhsh, P., Demirel, H. 4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations. Vis Comput 36, 951–965 (2020). https://doi.org/10.1007/s00371-019-01705-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01705-7

Keywords

Search

Navigation