Skip to main content
SpringerLink
Log in

Deep cross feature adaptive network for facial emotion classification

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel CNN-based model named as Deep Cross Feature Adaptive Network (DCFA-CNN) for facial expression recognition. The proposed DCFA-CNN model holds two major components: shape feature (ShFeat) block and texture feature (TexFeat) block, respectively. The ShFeat block is responsible to extract high-level responses, which leads to discriminate features from different expressive regions, while TexFeat block leads to hold micro/minute variations which defines structural differences in the expressive regions. Moreover, DCFA-CNN embedded a two-branch cross-relationship to collect information of ShFeat and TexFeat block. These different responses boost discriminability of the network by incorporating complementary features. The effectiveness of the proposed DCFA-CNN is evaluated extensively with four datasets: CK+, MUG, ISED and OULU-CASIA, over single-domain subject independent and cross-domain ethnicity independent experimental setups. The experimental results show a significant improvement of 21.8%, 21.55% and 6.43%, 17.9% as compared with MobileNet for 6- and 7-classes over ISED and OULU-CASIA. The extensive ablation experiments have done to validate the role of each module in DCFA-CNN framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Ekman, P.: Basic emotions. Handbook of cognition and emotion vol. 98, pp. 45–60 (1999)

  2. Hong, H., Neven, H., Von der Malsburg, C.: Online facial expression recognition based on personalized galleries. Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 354–359 (1998)

  3. Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans. Image Process. 16(1), 172–187 (2006)

    Article  MathSciNet  Google Scholar 

  4. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  5. Jabid, T., Kabir, Md.H., Chae, O.: Robust facial expression recognition based on local directional pattern. ETRI J. 32(5), 784–794 (2010)

    Article  Google Scholar 

  6. Rivera, A.R., Castillo, J.R., Chae, O.: Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2012)

    Article  MathSciNet  Google Scholar 

  7. Rivera, A.R., Castillo, J.R., Chae, O.: Local directional texture pattern image descriptor. Pattern Recogn. Lett. 51, 94–100 (2015)

    Article  Google Scholar 

  8. Ryu, B., Rivera, A.R., Kim, J.: Local directional ternary pattern for facial expression recognition. IEEE Trans. Image Process. 26(12), 6006–6018 (2017)

    Article  MathSciNet  Google Scholar 

  9. Mandal, M., Verma, M., Mathur, S., Vipparthi, S.K., Murala, S., Kumar, D.K.: Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition. IET Image Proc. 13(5), 850–861 (2019)

    Article  Google Scholar 

  10. Verma, M., Sexena, P., Vipparthi, S. K., Santosh Singh, G:. QUEST: Quadriletral senary bit pattern for facial expression recognition, 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1498–1503 (2018)

  11. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  12. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014)

  13. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9 (2015)

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)

  15. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861 (2017)

  16. Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans. Cybern. 48(1), 103–114 (2016)

    Article  Google Scholar 

  17. Kim, Y., Yoo, B., Kwak, Y., Choi, C., Kim, J.: Deep generative-contrastive networks for facial expression recognition, arXiv preprint arXiv:1703.07140 (2017)

  18. Verma, M., Bhui, J. K., Vipparthi, S. K., Singh, G.: EXPERTNet: Exigent Features Preservative Network for Facial Expression Recognition, Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–8 (2018)

  19. Zhang, S., Pan, X., Cui, Y., Zhao, X., Liu, L.: Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7, 32297–32304 (2019)

    Article  Google Scholar 

  20. Verma, Monu, SK, Vipparthi, Singh, G.: Hinet: Hybrid inherited feature learning network for facial expression recognition. IEEE Lett. Comput. Soc. 2(4), 36–39 (2019)

    Article  Google Scholar 

  21. Georgescu, M., Ionescu, R.T., Popescu, M.: Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7, 64827–64836 (2019)

    Article  Google Scholar 

  22. Liang, D., Liang, H., Yu, Z., Zhang, Y.: Deep convolutional BiLSTM fusion network for facial expression recognition. Vis. Comput. 36(3), 499–508 (2020)

    Article  Google Scholar 

  23. Minaee, S., Abdolrashidi, A.: Deep-emotion: Facial expression recognition using attentional convolutional network, arXiv preprint arXiv:1902.01019, (2019)

  24. Wang, K., Peng, X., Yang, J., Meng, D., Qiao, Y.: Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans. Image Process. 29, 4057–4069 (2020)

    Article  Google Scholar 

  25. Verma, M., Saxena, P., Vipparthi, S. K., Singh, G., Girdhari , Nagar, S. K.: DEFINET: portable CNN network for facial expression recognition, ICT for Competitive Strategies: Proceedings of 4th international conference on information and communication technology for competitive strategies (ICTCS), vol. 217 (2020)

  26. Xie, S., Hu, H.: Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans. Multimed. 2(1), 211–220 (2018)

    Article  Google Scholar 

  27. Verma, M., Vipparthi, S.K., Singh, G., Murala, S.: LEARNet: Dynamic imaging network for micro expression recognition. IEEE Trans. Image Process. 29, 1618–1627 (2019)

    Article  MathSciNet  Google Scholar 

  28. Verma, M., Vipparthi, S.K., Singh, G.: AffectiveNet: affective-motion feature learning for micro expression recognition. IEEE Multimed. 28, 17–27 (2020)

    Article  Google Scholar 

  29. Schindler, A., Lidy, T., Rauber, A.: Comparing shallow versus deep neural network architectures for automatic music genre classification. FMT, 17–21 (2016)

  30. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, 2010 IEEEE computer society conference on computer vision and pattern recognition-workshops, pp. 94–101 (2010)

  31. Aifanti, N., Papachristou, C., Delopoulos, A.: The MUG facial expression database, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1–4 (2010)

  32. Happy, S.L., Patnaik, P., Routray, A., Guha, R.: The indian spontaneous expression database for emotion recognition. IEEE Trans. Affect. Comput. 8(1), 131–142 (2015)

    Article  Google Scholar 

  33. Zhao, G., Huang, X., Taini, M., Li, S.Z., PietikäInen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Article  Google Scholar 

  34. Verma, M., Vipparthi, S. K., Singh, G.: Non-Linearities Improve OrigiNet based on Active Imaging for Micro Expression Recognition, arXiv preprint arXiv:2005.07991 (2020)

Download references

Author information

Authors and Affiliations

  1. Geethanjali College of Engineering and Technology, Medchal, India

    A. Hariprasad Reddy & Kamakshaiah Kolli

  2. Sri Satya Sai University of Technology and Medical Sciences, Sehore, India

    Y. Lakshmi Kiran

Authors
  1. A. Hariprasad Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kamakshaiah Kolli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Y. Lakshmi Kiran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Hariprasad Reddy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reddy, A.H., Kolli, K. & Kiran, Y.L. Deep cross feature adaptive network for facial emotion classification. SIViP 16, 369–376 (2022). https://doi.org/10.1007/s11760-021-01941-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01941-2

Keywords

Search

Navigation