Skip to main content

Advertisement

SpringerLink
Log in

Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information

  • Published:
Computing Aims and scope Submit manuscript

Abstract

The boom of big data in education has provided an unrivalled opportunity for educators to evaluate the learners’ cognitive state. However, most existing cognitive state analysis methods focus on attention, ignoring the roles of emotion in human learning. Therefore, this study proposes an emotion-sensitive learning cognitive state analysis framework, which automatically estimates the learners’ attention based on head pose and emotion based on facial expression in a non-invasive way. The proposed framework includes two modules. In the first module, a multi-task learning implementation with a cascaded convolutional neural network (CNN) is presented for face detection, landmark location, and head pose estimation simultaneously. The located landmarks are used to align the faces as the preprocessing step for the facial expression analysis. The estimated head pose and landmarks are used to recognize the visual focus of attention of the learner. In the second module, an expression intensity ranking CNN is proposed to recognize the facial expression and evaluate its intensity using ordinal information of the sequences. Then, the learners’ emotions are estimated based on the facial expression. Experimental results show that this method can estimate a learner’s attention and emotion with correctness rates of 79.5% and 88.6%, respectively. The results obtained suggest that the method has strong potential as an alternative method for analyzing emotion-sensitive learning cognitive state.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ashby FG, Isen AM, Turken AU (1999) A neuropsychological theory of positive affect and its influence on cognition. Psychol Rev 106(3):529–550

    Article  Google Scholar 

  2. Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose-application in an e-learning environment. Multimed Tools Appl 41(3):469–493

    Article  Google Scholar 

  3. Batista JC, Albiero V, Bellon ORP, Silva L (2017) Aumpnet: simultaneous action units detection and intensity estimation on multipose facial images using a single convolutional neural network. In: IEEE international conference on automatic face and gesture recognition, pp 866–871

  4. Chen D, Hu Y, Wang L, Zomaya AY, Li X (2017) H-PARAFAC: Hierarchical parallel factor analysis of multidimensional big data. IEEE Trans Parallel Distrib Syst 28(4):1091–1104

    Article  Google Scholar 

  5. Chen J, Luo N, Liu Y, Liu L, Zhang K, Kolodziej J (2016) A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1–2):215–233

    Article  MathSciNet  Google Scholar 

  6. Chen S, Zhang C, Dong M, Le J, Rao M (2017) Using ranking-CNN for age estimation. In: IEEE conference on computer vision and pattern recognition, pp 742–751

  7. Chu W, Ghahramani Z (2004) Gaussian processes for ordinal regression. J Mach Learn Res 6(3):1019–1041

    MathSciNet  MATH  Google Scholar 

  8. Dong Y, Huang C, Liu W (2014) RankCNN: when learning to rank encounters the pseudo preference feedback. Comput Stand Interfaces 36(3):554–562

    Article  Google Scholar 

  9. Ekman P (1978) Facial action coding system. A technique for the measurement of facial action. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  10. Fan Y, Shen D, Davatzikos C (2006) Detecting cognitive states from FMRI images by machine learning and multivariate classification. In: Conference on computer vision and pattern recognition workshop, 2006. CVPRW ’06, p 89

  11. Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions, vol 7432. Springer, Berlin

    Google Scholar 

  12. Ke H, Chen D, Shah T, Liu X, Zhang X, Zhang L, Li X (2018) Cloud-aided online EEG classification system for brain healthcare: a case study of depression evaluation with a lightweight CNN. Software: Practice and Experience

  13. Koelstra S, Pantic M (2008) Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics. IEEE Trans Med Imaging 18(8):712–721

    Google Scholar 

  14. Köstinger M, Wohlhart P, Roth PM, Bischof H (2012) Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: IEEE international conference on computer vision workshops, pp 2144–2151

  15. Li S, Deng W (2018) Deep facial expression recognition: A survey. arXiv:1804.08348

  16. Liao CT, Chuang HJ, Lai SH (2012) Learning expression kernels for facial expression intensity estimation. In: IEEE international conference on acoustics, speech and signal processing, pp 2217–2220

  17. Littlewort G, Bartlett MS, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24(6):615–625

    Article  Google Scholar 

  18. Liu M, Li S, Shan S, Wang R, Chen X (2014) Deeply learning deformable facial action parts model for dynamic expression analysis. Springer, Berlin

    Google Scholar 

  19. Liu Y, Chen J, Zhang M, Rao C (2018) Student engagement study based on multi-cue detection and recognition in an intelligent learning environment. Multimed Tools Appl 77:28749–28775

    Article  Google Scholar 

  20. Liu Y, Wang L, Li W (2017) Emotion analysis based on facial expression recognition in virtual learning environment. Int J Comput Commun Eng 6:49–56

    Article  Google Scholar 

  21. Lucey P, Cohn JF, Prkachin KM, Solomon PE (2011) Painful data: the UNBC-McMaster shoulder pain expression archive database. In: IEEE international conference on automatic face and gesture recognition, pp 57–64

  22. Mehrabian A, Wiener M (1967) Decoding of inconsistent communications. J Pers Soc Psychol 6(1):109–114

    Article  Google Scholar 

  23. Odobez JM, Ba S (2007) A cognitive and unsupervised map adaptation approach to the recognition of the focus of attention from head pose. In: IEEE international conference on multimedia and expo, pp 1379–1382

  24. Ranjan R, Patel VM, Chellappa R (2016) Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41:121–135

    Article  Google Scholar 

  25. Shen L, Wang M, Shen R (2009) Affective e-learning: using “emotional” data to improve learning in pervasive learning environment. J Educ Technol Soc 12(2):176–189

    Google Scholar 

  26. Shen WY, Lin HT (2017) Active sampling of pairs and points for large-scale linear bipartite ranking. J Mach Learn Res 29:388–403

    Google Scholar 

  27. Siau K, Sheng H, Nah FH (2006) Use of a classroom response system to enhance classroom interactivity. IEEE Trans Educ 49(3):398–403

    Article  Google Scholar 

  28. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations

  29. Tang Y, Chen D, Wang L, Zomaya AY, Chen J, Liu H (2018) Bayesian tensor factorization for multi-way analysis of multi-dimensional EEG. Neurocomputing 318:162–174

    Article  Google Scholar 

  30. Tian YL, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  31. Wei C, Keerthi SS (2005) New approaches to support vector ordinal regression. In: International conference on machine learning, pp 145–152

  32. Yang P, Liu Q, Metaxas DN (2010) Rankboost with l1 regularization for facial expression recognition and intensity estimation. In: IEEE international conference on computer vision, pp 1018–1025

  33. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: International conference on automatic face and gesture recognition, pp 211–216

  34. Yun WH, Lee D, Park C, Kim J (2015) Automatic engagement level estimation of kids in a learning environment. Int J Mach Learn Comput 5(2):148–152

    Article  Google Scholar 

  35. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  36. Zhao R, Gan Q, Wang S, Ji Q (2016) Facial expression intensity estimation using ordinal information. In: Computer vision and pattern recognition, pp 3466–3474

  37. Zhao X, Liang X, Liu L, Li T, Han Y, Vasconcelos N, Yan S (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision, pp 425–442

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (Grant No. 2018YFB1004504), the National Natural Science Foundation of China (No. 61772380, No. 61273063), Foundation for Innovative Research Groups of Hubei Province (No. 2017CFA007), and Hubei Province Technological Innovation Major Project and Research Funds of CCNU from the Colleges’ Basic Research and Operation of MOE (Grant No. CCNU19Z02002).

Author information

Authors and Affiliations

  1. National Engineering Research Center for E-Learning, Central China Normal University, Wuhan, China

    Ruyi Xu, Jingying Chen, Jiaxu Han, Lei Tan & Luhui Xu

  2. National Engineering Laboratory for Educational Big Data, Central China Normal University, Wuhan, China

    Ruyi Xu, Jingying Chen, Jiaxu Han, Lei Tan & Luhui Xu

Authors
  1. Ruyi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaxu Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luhui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingying Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, R., Chen, J., Han, J. et al. Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information. Computing 102, 765–780 (2020). https://doi.org/10.1007/s00607-019-00722-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-019-00722-7

Keywords

Mathematics Subject Classification

Search

Navigation