Improving face verification using facial marks and deep CNN: IARPA Janus benchmark-A

https://doi.org/10.1016/j.imavis.2020.104020Get rights and content

Highlights

  • Face verification with soft biometric based matching approach

  • Unconstrained dataset with larger variations collected from 500 subjects

  • Evaluations on FERET, CFM, Mugshot, and IJB-A datasets

  • Performance improvement with Facial mark and VGG-face based fusion matching

Sidra Riaz received her BS degree in Telecom Engineering from NUCES-FAST in 2011. She received her MS and PhD degrees in Computer Engineering from Chosun University in 2013 and Sogang University in 2020, respectively. She is currently working as an AI developer in BluePrint Company, Korea. Ms. Sidra Riaz is the recipient of National ICT R&D Merit Scholarship, Global IT Scholarship, and Seoul Honorary Citizenship Awards. Her research interests include deep learning, computer vision, and pattern recognition.

Unsang Park received the BS and MS degrees from the Department of Materials Engineering, Hanyang University, South Korea, in 1998 and 2000, respectively. He received the MS and Ph.D. degrees from the Department of Computer Science and Engineering, Michigan State University, in 2004 and 2009, respectively. From 2012, he is an assistant professor in the Department of Computer Science and Engineering at Sogang University. His research interests include pattern recognition, image processing, computer vision, and machine learning.

Prem Natarajan is Michael Keston Executive Director of ISI, a vice dean of the USC Viterbi School of Engineering and a professor of computer science. At ISI, he leads managerial and technical directions Institute-wide, including research, development and the MOSIS electronic chip brokerage. He also heads teams in his areas of expertise: novel approaches to face, character, handwriting and speech recognition, along with other deep learning and natural language processing directions.

Abstract

Face verification performance by human brain has been shown to be much better than most of the state-of-the-art approaches in computer vision. Performance improvement of automated face recognition (FR) systems that may equal or surpass the human intellect is the key goal of this research. We present our face verification system using facial mark (FM) combined with deep convolutional neural network (DCNN) approach to improve the overall FR accuracy. We propose to use FM (e.g., scars, moles and freckles) for face matching in the wild where the FM detection is performed on mean faces as well as affine aligned normalized facial images. The FR experiments are carried out on IARPA Janus Benchmark-A (IJB-A) dataset which includes real-world unconstrained images from 500 subjects. The IJB-A datasets includes full pose, expression, and illumination variations which are much harder than traditional FERET and Mugshot datasets. We evaluated the average FR performance using a weighted score-level fusion of FM and DCNN based recognition methods. The experimental evaluations on FERET, CFM and Mugshot datasets show higher performances than state-of-the-art FM approaches with 99.23%, 94.64% and 97.86% accuracies in Rank-1 evaluations, respectively. Our FR performance of FM + DCNN (86.46% in TAR, 91.23% in Rank-1, 96.57% in Rank-5, 98.65% in Rank-10) is shown to be higher than state-of-the-art (83.80% in TAR@1%FAR, 90.30% in Rank-1, 96.5% in Rank-5, and 97.7% in Rank-10). Experimental results after fusion of FM + DCNN on IJB-A dataset show 2.66% FR performance improvement from the DCNN only recognition in terms of TAR@1%FAR.

Introduction

Face verification is one of the most challenging topics in computer vision and has been the most actively researched area over the past few decades [3], [4], [7], [8]. The key objective of face verification systems is to determine if the two images or videos belong to the same person. Many face recognition algorithms have been presented by researchers that work well under the controlled settings. However, we are yet anticipating to build a system that is invariant to variations in pose, illumination, expression, and occlusion, showing comparable accuracy as that of a human. To deal with the aforementioned challenges, we developed a fusion based face recognition method using soft biometric, i.e. facial mark (FM) [1], [5], [8] and deep convolutional neural network (DCNN) [31] methods to achieve higher verification accuracy. The proposed system is evaluated on the unconstrained IJB-A dataset [9] (more than 5000 probe and gallery images from 500 subjects including various pose, illumination, expression, aging, image resolution, and occlusion).

In face verification, FM serves significantly in providing valuable information in distinguishing identities [7] where a face image is occluded or captured in an off-frontal pose. The FM are the embedded traits in an individual's face that can-not be fully distinctive by themselves. However, this biometric information could be incorporated in a face verification system by utilizing the FM based matching results as a complementary information. We presented an automatic FM detection module that takes a 2D input image and performs: i) facial feature (nose, eye, mouth) localization, ii) user-specific mask construction, iii) blob detection, and iv) morphological operations. More details about FM detection are provided in Section 3.1. Recent advancements in computer vision approaches show that DCNN based approaches perform well for face verification task. The DCNN is a biologically inspired mathematical model that is capable of achieving top performance in various tasks of computer vision, including object classification [11], face detection [12], [13], facial landmark detection [14], face alignment [15] and face verification [16]. The DCNN has a powerful learning capability and consists of both training and testing stages. In our FR system, the training of DCNN is performed by using CASIA-WebFace dataset.1 Given a test dataset (IJB-A) image, a similarity score is computed based on the DCNN features and the learned metrics [9], [17]. The score-level fusion of DCNN and FM based recognition method achieves accuracy improvement over the conventional schemes, whose detail will be explained in the upcoming sections.

Following are the major contributions of this paper.

  • Unconstrained Dataset: Our face mark (FM) based approach is tested on unconstrained IJB-A datasets which gives us larger variations in pose, lighting, expression, aging, occlusion, and illumination. The images in IJB-A dataset are collected from 500 subjects which include wider geographic variations of the subjects in a fully unconstrained environment. Also, the images are collected at a different resolution. The IJB-A dataset is much harder than the FERET, Mugshot and FG-NET datasets used in the previous works for mark detection and face recognition.

  • FM based Matching Score: We developed a score computation scheme for FM based matching. Each mark detected on the probe image is compared with all the marks in the gallery image. Two marks on Probe and Gallery images are considered a match when the matching distance (selected empirically) between them is less than the threshold (τ = 13). We selected the matching threshold empirically. The score is computed as the inverse of the distance and sum of number of matched marks.

  • Fusion of Soft Biometrics and VGG Descriptors: For performance improvement, we combined the two face recognition approaches using FM and VGG based matching. The fusion is performed by the weighted score sum method. The overall face recognition performance of our approach is better than the previous approaches in FM which report their results on comparatively easier datasets, i.e. FERET and Mugshot datasets.

  • Performance Evaluations using IJB-A, FERET, and Mugshot Datasets: For performance evaluations, we used the IJB-A test dataset which is highly unconstrained to the best of our knowledge and haven't been evaluated yet in the FM based FR papers. Thus our findings in face recognition d omain using IJB-A dataset may serve as a baseline for further explorations and studies. Furthermore, for a fair comparison with state-of-the-art FM approaches, we have evaluated our FM + DCNN based matching method on FERET and Mugshot datasets. The comparisons are provided in the experimental result section.

The rest of the paper is organized as follows. In Section 2, we review the previous studies in face identification utilizing soft biometrics and DCNN based matching. Section 3 addresses different components of the proposed approach of face verification by using FM combined with DCNN method. In Section 4, we provide experimental results in details reporting the fusion performance of FM + DCNN based face matching. The experimentations are performed on FERET, Mughsot, and IJB-A dataset which is described in the subsection of Section 4. In Section 5, we conclude our work by providing an insight into the future research direction.

Section snippets

Related work

Soft biometrics for face recognition is a contemporary topic in computer vision research [2], [7], [8], [9], [10] and has recently gained wide interest for various reasons, such as the need for higher reliability in biometric systems [6] or quick filtering of candidate matches. In various recognition schemes, a number of studies were carried out on exploiting soft biometrics [1], [5], [8] such as gender, ethnicity, skin texture, scars, moles and tattoos [1], [5], [10]. Soft biometrics are used

Proposed face verification method

Our identity verification scheme consists of two subcomponents such as, (1) facial mark (FM) and (2) deep convolutional neural network (DCNN). Each of the facial recognition sub-component is applied on a large IJB-A test dataset for computing the similarity scores and then weighted score level fusion is performed to generate the final results. Each component is described in details in the following sub-sections.

Experimental results and analysis

The FM only and FM + DCNN based matching experiments are performed on IJB-A datasets. The comparison with state-of-the-art is also provided. More details about our experiments and IJB-A dataset are given in the following subsections.

Conclusion

In this paper, we presented our FM based matching method for face verification. The VGG face descriptors are used for matching the affine aligned and mean face images. For accuracy improvement, we combine the FM and DCNN based matching approaches at various matching thresholds. The matching scores obtained by two verification schemes are fused together by a weighted score sum method. The fusion weights were selected empirically. Further, the matching threshold for FM based face verification was

Declaration of Competing Interest

The authors declare no conflicts of interest.

Acknowledgment

This work was supported by the ICT R&D By the Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government (MSIT) [Project Number : 2020-0-00113, Project Name : Development of data augmentation technology by using heterogeneous information and data fusions], and the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA contract number 2014-14071600011. The views and conclusions

Declaration of Competing Interest

The author(s) declare(s) that there is no conflict of interest regarding the publication of this paper.

References (38)

  • J.C. Chen et al.

    “Unconstrained Face Verification Using Deep CNN Features,” 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY

    (2016)
  • U.R. Gogoi et al.

    Facial mole detection: an approach towards face identification

    Procedia Computer Science

    (2015)
  • A.K. Jain et al.

    Facial marks: Soft biometric for face recognition

  • J.L. Dugelay

    Keynote speech 1: Bag of visual soft biometrics for person identification: New trends and challenges

  • U. Park et al.

    Face matching and retrieval using soft biometrics

    IEEE Transactions on Information Forensics and Security

    (2010)
  • C.G. Zeinstra et al.

    Forensic face recognition as a means to determine strength of evidence: A survey

    Forensic Science Review

    (2018)
  • A.K. Jain et al.

    Integrating faces, fingerprints, and soft biometric traits for user recognition

  • K. Niinuma et al.

    Soft biometric traits for continuous user authentication

  • A. Dantcheva et al.

    What Else does your biometric data reveal? A survey on soft biometrics

    in IEEE Transactions on Information Forensics and Security

    (March 2016)
  • A.K. Jain et al.

    Soft biometric traits for personal recognition systems, Biometric Authentication

    (2004)
  • B.F. Klare

    “Pushing the Frontiers of Unconstrained Face Detection and Recognition: IARPA Janus Benchmark a,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA

    (2015)
  • N. Kumar et al.

    Describable visual attributes for face verification and image search

  • H. Jang et al.

    Object Classification Using CNN for Video Traffic detection system, 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), Mokpo

    (2015)
  • Haoxiang Li

    A Convolutional Neural Network Cascade for Face Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2015)
  • S.-S. Farfade et al.

    “Multi-view Face Detection Using Deep Convolutional Neural Networks,” Computer Vision and Pattern Recogniton (CVPR)

    (2015)
  • Yi Sun et al.

    Deep Convolutional Network Cascade for Facial Point Detection, Computer Vision and Pattern Recognition (CVPR), IEEE Conference on, IEEE

    (2013)
  • Xuehan Xiong et al.

    “Supervised descent method and its applications to face alignment,” Computer Vision and Pattern Recognition (CVPR), 2013 IEEE conference on. IEEE

    (2013)
  • A. Bansal et al.

    The Do’s and Don’ts for CNN-based Face Verification, Computer Vision and Pattern Recognition (CVPR), arXiv. bs/1705.07426

    (2017)
  • Dahua Lin et al.

    "Recognize High Resolution Faces: From Macrocosm to Microcosm," 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)

    (2006)
  • Cited by (0)

    View full text