Skip to main content
SpringerLink
Log in

Facial asymmetry-based feature extraction for different applications: a review complemented by new advances

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Facial feature extraction (FFE) is considered as a challenging area for computer vision and artificial intelligence research community. There are numerous application domains of face recognition such as demographic classification and facial disease classification. In last few years, numerous feature extraction approaches are proposed. This review focuses on studies that exclusively use facial asymmetry as one of the main subject-specific facial characteristics for FFE. This paper provides a review about the related research conducted in the past two decades. First, we summarize the conventional FFE approaches and their main algorithms. Application of deep networks to facial asymmetry based FFE approaches is then presented. Multi-network deep models suitable for asymmetry-based FFE for different applications are also focused in this review. We presented the details about the publicly available face datasets, evaluation metrics, and comparison of the state-of-the-art results is also presented. The directions of asymmetry-based FFE for future research is also presented to provide an awareness about the existing and future trends. This review is presented to provide directions and ideas for future research in the field of facial asymmetry.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Bartneck Christoph, Lyons Michael J (2007) Hci and the face: towards an art of the soluble. In: International conference on human–computer interaction. Springer, pp 20–29

  • Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346–359

    Article  Google Scholar 

  • Bianco S (2017) Large age-gap face verification by feature injection in deep networks. Pattern Recogn Lett 90:36–42

    Article  Google Scholar 

  • Bit-biometrics ideal test. Accessed 11 Sept 2018

  • Bobulski J (2015) Two-dimensional hidden markov models in road signs recognition. In: Image processing and communications challenges, vol 6, pp 3–12. Springer

  • Bobulski J (2016) Face recognition method with two-dimensional hmm. In: Proceedings of the 9th international conference on computer recognition systems CORES 2015. Springer, pp 317–325

  • Brachmann A, Redies C (2016) Using convolutional neural network filters to measure left-right mirror symmetry in images. Symmetry 8(12):144

    Article  MathSciNet  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2016) Local gradient hexa pattern: a descriptor for face recognition and retrieval. IEEE Trans Circuits Syst Video Technol 28(1):171–180

    Article  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2017) Local directional gradient pattern: a local descriptor for face recognition. Multimed Tools Appl 76(1):1201–1216

    Article  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2017) Local quadruple pattern: a novel descriptor for facial image recognition and retrieval. Comput Electric Eng 62:92–104

    Article  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2018) Centre symmetric quadruple pattern: a novel descriptor for facial image recognition and retrieval. Pattern Recogn Lett 115:50–58

    Article  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2019) R-theta local neighborhood pattern for unconstrained facial image recognition and retrieval. Multimed Tools Appl 78(11):14799–14822

    Article  Google Scholar 

  • Chakraborty S, Singh SK, Chakraborty P (2019) Cascaded asymmetric local pattern: a novel descriptor for unconstrained facial image recognition and retrieval. Multimed Tools Appl 78(17):25143–25162

    Article  Google Scholar 

  • Chen B-C, Chen C-S, Hsu WH (2015) Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Trans Multimed 17(6):804–815

    Article  Google Scholar 

  • Chenfei X, Liu Q, Ye M (2017) Age invariant face recognition and retrieval by coupled auto-encoder networks. Neurocomputing 222:62–71

    Article  Google Scholar 

  • Cheng G, Zhou P, Han J (2016) Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images. IEEE Trans Geosci Remote Sens 54(12):7405–7415

    Article  Google Scholar 

  • Chollet F (2015) Keras

  • Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 6:681–685

    Article  Google Scholar 

  • Dubey SR, Roy SK, Chakraborty S, Mukherjee S, Chaudhuri BB (2019) Local bit-plane decoded convolutional neural network features for biomedical image retrieval. Neural Comput Appl 1–13

  • Ercan I, Ozdemir ST, Etoz A, Sigirli D, Shane Tubbs R, Loukas M, Guney I (2008) Facial asymmetry in young healthy subjects evaluated by statistical shape analysis. J Anat 213(6):663–669

    Article  Google Scholar 

  • Fabian Benitez-Quiroz C, Srinivasan R, Martinez AM (2016) Emotionet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5562–5570

  • Face++ toolkit. Accessed 3 Jan 2017

  • FERET database

  • Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst

  • Gower JC (1975) Generalized procrustes analysis. Psychometrika 40(1):33–51

    Article  MathSciNet  MATH  Google Scholar 

  • Graham JH, Özener B (2016) Fluctuating asymmetry of human populations: a review. Symmetry 8(12):154

    Article  MathSciNet  Google Scholar 

  • Gutta S, Wechsler H (2004) Face recognition using asymmetric faces. In: International conference on biometric authentication. Springer, pp 162–168

  • Harguess J, Aggarwal JK (2011) Is there a connection between face symmetry and face recognition? In: CVPR 2011 WORKSHOPS. IEEE, pp 66–73

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • House WE (1985) Facial nerve grading system. Otolaryngol Head Neck Surg 93:184–193

    Article  Google Scholar 

  • Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  • Huiskes MJ, Lew MS (2008) The mir flickr retrieval evaluation. In: Proceedings of the 1st ACM international conference on multimedia information retrieval. ACM, pp 39–43

  • Jonathon PP, Flynn PJ, Bowyer KW, Vorder BRW, Grother PJ, Quinn GW, Pruitt M (2011) Distinguishing identical twins by face recognition. In: Face and gesture 2011. IEEE, pp 185–192

  • Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580). IEEE, pp 46–53

  • Kaynak MN, Zhi Q, Cheok AD, Sengupta K, Chung KC (2001) Audio-visual modeling for bimodal speech recognition. In: 2001 IEEE international conference on systems, man and cybernetics. e-systems and e-man for cybernetics in cyberspace (Cat. No. 01CH37236), vol 1. IEEE, pp 181–186

  • Khotanzad A, Hong YH (1990) Invariant image recognition by zernike moments. IEEE Trans Pattern Anal Mach Intell 12(5):489–497

    Article  Google Scholar 

  • Kim H, Kim S, Kim Y, Park K (2015) A smartphone-based automatic diagnosis system for facial nerve palsy. Sensors 15(10):26756–26768

    Article  Google Scholar 

  • Ko BC, Lee EJ, Nam JY (2016) Genetic algorithm based filter bank design for light convolutional neural network. Adv Sci Lett 22(9):2310–2313

    Article  Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • Kubanek M, Bobulski J (2017) The use of hidden markov models to verify the identity based on facial asymmetry. EURASIP J Image Video Process 2017(1):45

    Article  Google Scholar 

  • Latif A, Rasheed A, Sajid U, Ahmed J, Ali N, Ratyal NI, Zafar B, Dar Saadat H, Sajid M, Khalil T (2019) Content-based image retrieval and feature extraction: a comprehensive review. Math Problems Eng 2019

  • Le Hoang Ngan T, Seshadri K, Luu K, Savvides M (2015) Facial aging and asymmetry decomposition based approaches to identification of twins. Pattern Recogn 48(12):3843–3856

    Article  Google Scholar 

  • LeCun Y, Bottou L, Bengio Y, Haffner P et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  • Lemley J, Bazrafkan S, Corcoran P (2017) Smart augmentation learning an optimal data augmentation strategy. Ieee Access 5:5858–5869

    Article  Google Scholar 

  • Liu Y, Schmidt KL, Cohn JF, Mitra S (2003) Facial asymmetry quantification for expression invariant human identification. Comput Vis Image Underst 91(1–2):138–159

    Article  Google Scholar 

  • Liu L, Xiong C, Zhang H, Niu Z, Wang M, Yan S (2015) Deep aging face verification with large gaps. IEEE Trans Multimed 18(1):64–75

    Article  Google Scholar 

  • Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110

    Article  Google Scholar 

  • Lu J, Liong VE, Zhou J (2017) Simultaneous local binary feature learning and encoding for homogeneous and heterogeneous face recognition. IEEE Trans Pattern Anal Mach Intell 40(8):1979–1993

    Article  Google Scholar 

  • Ming-Kuei H (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8(2):179–187

    Article  MATH  Google Scholar 

  • Mitra S, Lazar NA, Liu Y (2007) Understanding the role of facial asymmetry in human face identification. Stat Comput 17(1):57–70

    Article  MathSciNet  Google Scholar 

  • Moayedikia A, Ong K-L, Boo YL, Yeoh WGS, Jensen R (2017) Feature selection for high dimensional imbalanced class data using harmony search. Eng Appl Artif Intell 57:38–49

    Article  Google Scholar 

  • Otto C, Han H, Jain A (2012) How does aging affect facial components? In: European conference on computer vision. Springer, pp 189–198

  • Panis G, Lanitis A, Tsapatsoulis N, Cootes TF (2016) Overview of research on facial ageing using the fg-net ageing database. Iet Biometrics 5(2):37–46

    Article  Google Scholar 

  • Parkhi OM, Vedaldi A, Zisserman A et al (2015) Deep face recognition. BMVC 1:6

    Google Scholar 

  • Pesaresi M, Benediktsson JA (2001) A new approach for the morphological segmentation of high-resolution satellite imagery. IEEE Trans Geosci Remote Sens 39(2):309–320

    Article  Google Scholar 

  • Powers DM (2011) Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation

  • Rajaraman S, Antani SK, Poostchi M, Kamolrat Silamut MA, Hossain RJ, Maude SJ, Thoma GR (2018) Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images. PeerJ 6:e4568

    Article  Google Scholar 

  • Ricanek K, Tesafaye T (2006) Morph: a longitudinal image database of normal adult age-progression. In: 7th International conference on automatic face and gesture recognition (FGR06). IEEE, pp 341–345

  • Sajid M, Taj IA, Bajwa UI, Ratyal NI (2016) The role of facial asymmetry in recognizing age-separated face images. Comput Electric Eng 54:255–270

    Article  Google Scholar 

  • Sajid M, Taj IA, Bajwa UI, Ratyal NI (2018) Facial asymmetry-based age group estimation: role in recognizing age-separated face images. J Forensic Sci 63(6):1727–1749

    Article  Google Scholar 

  • Sajid M, Shafique T, Manzoor S, Iqbal F, Talal H, Qureshi US, Riaz I (2018) Demographic-assisted age-invariant face recognition and retrieval. Symmetry 10(5):148

    Article  Google Scholar 

  • Sajid M, Shafique T, Baig M, Riaz I, Amin S, Manzoor S (2018) Automatic grading of palsy using asymmetrical facial features: a study complemented by new solutions. Symmetry 10(7):242

    Article  Google Scholar 

  • Sajid M, Shafique T, Riaz I, Imran M, Baig MJA, Baig S, Manzoor S (2018) Facial asymmetry-based anthropometric differences between gender and ethnicity. Symmetry 10(7):232

    Article  Google Scholar 

  • Sforza C, Laino A, Grandi G, Pisoni L, Ferrario VF (2010) Three-dimensional facial asymmetry in attractive and normal people from childhood to young adulthood. Symmetry 2(4):1925–1944

    Article  Google Scholar 

  • Shrivastava A, Pfister T, Tuzel O, Susskind J, Wang W, Webb R (2017) Learning from simulated and unsupervised images through adversarial training. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2107–2116

  • Sivaramakrishnan R, Antani S, Xue Z, Candemir S, Jaeger S, Thoma GR (2017) Visualizing abnormalities in chest radiographs through salient network activations in deep learning. In: 2017 IEEE life sciences conference (LSC). IEEE, pp 71–74

  • Song I, Vong J, Yen NY, Diederich J, Yellowlees P (2013) Profiling bell’s palsy based on house-brackmann score. J Artif Intell Soft Comput Res 3(1):41–50

    Article  Google Scholar 

  • Suh K, Kim Y, Lee E (2016) Facial feature movements caused by various emotions: differences according to sex. Symmetry 8(9):86

    Article  MathSciNet  Google Scholar 

  • Suzuki K (2017) Overview of deep learning in medical imaging. Radiol Phys Technol 10(3):257–273

    Article  Google Scholar 

  • Texas 3d face recognition database (2010)

  • Tingting Yu, Junqian W, Lintai W, Yong X (2019) Three-stage network for age estimation. CAAI Trans Intell Technol 4(2):122–126

    Article  Google Scholar 

  • Turk M, Pentland A, Belhumeur P, Hespanha J (1991) Eigenfaces for recognition. J Cognit Neurosci

  • Verma NK, Hanmandlu M (2009) Additive and nonadditive fuzzy hidden markov models. IEEE Trans Fuzzy Syst 18(1):40–56

    Article  Google Scholar 

  • Wen Y, Li Z, Qiao Y (2016) Age invariant deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  • Zaidel DW, Hessamian M (2010) Asymmetry and symmetry in the beauty of human faces. Symmetry 2(1):136–149

    Article  Google Scholar 

  • Zhai H, Liu C, Dong H, Ji Y, Guo Y, Gong S (2015) Face verification across aging based on deep convolutional networks and local binary patterns. In: International conference on intelligent science and big data engineering. Springer, pp 341–350

  • Zhang H, Dong Z, Shu H (2010) Object recognition by a complete set of pseudo-zernike moment invariants. In: 2010 IEEE international conference on acoustics, speech and signal processing. IEEE, pp 930–933

  • Zhu C, Yan W, Cai X, Liu S, Li TH, Li G (2020) Neural saliency algorithm guide bi-directional visual perception style transfer. CAAI Trans Intell Technol 5(1):1–8

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Mirpur University of Science & Technology (MUST), Mirpur, 10250, AJK, Pakistan

    Muhammad Sajid

  2. Department of Software Engineering, Mirpur University of Science & Technology (MUST), Mirpur, 10250, AJK, Pakistan

    Nouman Ali & Saadat Hanif Dar

  3. Department of Computer Systems Engineering, Mirpur University of Science & Technology (MUST), Mirpur, 10250, AJK, Pakistan

    Naeem Iqbal Ratyal

  4. Department of Computer Science, Government College University, Faisalabad, 38000, Pakistan

    Bushra Zafar

Authors
  1. Muhammad Sajid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nouman Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Naeem Iqbal Ratyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saadat Hanif Dar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bushra Zafar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nouman Ali.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sajid, M., Ali, N., Ratyal, N.I. et al. Facial asymmetry-based feature extraction for different applications: a review complemented by new advances. Artif Intell Rev 54, 4379–4419 (2021). https://doi.org/10.1007/s10462-021-10001-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-021-10001-7

Keywords

Search

Navigation