Improving face verification using facial marks and deep CNN: IARPA Janus benchmark-A
Introduction
Face verification is one of the most challenging topics in computer vision and has been the most actively researched area over the past few decades [3], [4], [7], [8]. The key objective of face verification systems is to determine if the two images or videos belong to the same person. Many face recognition algorithms have been presented by researchers that work well under the controlled settings. However, we are yet anticipating to build a system that is invariant to variations in pose, illumination, expression, and occlusion, showing comparable accuracy as that of a human. To deal with the aforementioned challenges, we developed a fusion based face recognition method using soft biometric, i.e. facial mark (FM) [1], [5], [8] and deep convolutional neural network (DCNN) [31] methods to achieve higher verification accuracy. The proposed system is evaluated on the unconstrained IJB-A dataset [9] (more than 5000 probe and gallery images from 500 subjects including various pose, illumination, expression, aging, image resolution, and occlusion).
In face verification, FM serves significantly in providing valuable information in distinguishing identities [7] where a face image is occluded or captured in an off-frontal pose. The FM are the embedded traits in an individual's face that can-not be fully distinctive by themselves. However, this biometric information could be incorporated in a face verification system by utilizing the FM based matching results as a complementary information. We presented an automatic FM detection module that takes a 2D input image and performs: i) facial feature (nose, eye, mouth) localization, ii) user-specific mask construction, iii) blob detection, and iv) morphological operations. More details about FM detection are provided in Section 3.1. Recent advancements in computer vision approaches show that DCNN based approaches perform well for face verification task. The DCNN is a biologically inspired mathematical model that is capable of achieving top performance in various tasks of computer vision, including object classification [11], face detection [12], [13], facial landmark detection [14], face alignment [15] and face verification [16]. The DCNN has a powerful learning capability and consists of both training and testing stages. In our FR system, the training of DCNN is performed by using CASIA-WebFace dataset.1 Given a test dataset (IJB-A) image, a similarity score is computed based on the DCNN features and the learned metrics [9], [17]. The score-level fusion of DCNN and FM based recognition method achieves accuracy improvement over the conventional schemes, whose detail will be explained in the upcoming sections.
Following are the major contributions of this paper.
- •
Unconstrained Dataset: Our face mark (FM) based approach is tested on unconstrained IJB-A datasets which gives us larger variations in pose, lighting, expression, aging, occlusion, and illumination. The images in IJB-A dataset are collected from 500 subjects which include wider geographic variations of the subjects in a fully unconstrained environment. Also, the images are collected at a different resolution. The IJB-A dataset is much harder than the FERET, Mugshot and FG-NET datasets used in the previous works for mark detection and face recognition.
- •
FM based Matching Score: We developed a score computation scheme for FM based matching. Each mark detected on the probe image is compared with all the marks in the gallery image. Two marks on Probe and Gallery images are considered a match when the matching distance (selected empirically) between them is less than the threshold (τ = 13). We selected the matching threshold empirically. The score is computed as the inverse of the distance and sum of number of matched marks.
- •
Fusion of Soft Biometrics and VGG Descriptors: For performance improvement, we combined the two face recognition approaches using FM and VGG based matching. The fusion is performed by the weighted score sum method. The overall face recognition performance of our approach is better than the previous approaches in FM which report their results on comparatively easier datasets, i.e. FERET and Mugshot datasets.
- •
Performance Evaluations using IJB-A, FERET, and Mugshot Datasets: For performance evaluations, we used the IJB-A test dataset which is highly unconstrained to the best of our knowledge and haven't been evaluated yet in the FM based FR papers. Thus our findings in face recognition d omain using IJB-A dataset may serve as a baseline for further explorations and studies. Furthermore, for a fair comparison with state-of-the-art FM approaches, we have evaluated our FM + DCNN based matching method on FERET and Mugshot datasets. The comparisons are provided in the experimental result section.
The rest of the paper is organized as follows. In Section 2, we review the previous studies in face identification utilizing soft biometrics and DCNN based matching. Section 3 addresses different components of the proposed approach of face verification by using FM combined with DCNN method. In Section 4, we provide experimental results in details reporting the fusion performance of FM + DCNN based face matching. The experimentations are performed on FERET, Mughsot, and IJB-A dataset which is described in the subsection of Section 4. In Section 5, we conclude our work by providing an insight into the future research direction.
Section snippets
Related work
Soft biometrics for face recognition is a contemporary topic in computer vision research [2], [7], [8], [9], [10] and has recently gained wide interest for various reasons, such as the need for higher reliability in biometric systems [6] or quick filtering of candidate matches. In various recognition schemes, a number of studies were carried out on exploiting soft biometrics [1], [5], [8] such as gender, ethnicity, skin texture, scars, moles and tattoos [1], [5], [10]. Soft biometrics are used
Proposed face verification method
Our identity verification scheme consists of two subcomponents such as, (1) facial mark (FM) and (2) deep convolutional neural network (DCNN). Each of the facial recognition sub-component is applied on a large IJB-A test dataset for computing the similarity scores and then weighted score level fusion is performed to generate the final results. Each component is described in details in the following sub-sections.
Experimental results and analysis
The FM only and FM + DCNN based matching experiments are performed on IJB-A datasets. The comparison with state-of-the-art is also provided. More details about our experiments and IJB-A dataset are given in the following subsections.
Conclusion
In this paper, we presented our FM based matching method for face verification. The VGG face descriptors are used for matching the affine aligned and mean face images. For accuracy improvement, we combine the FM and DCNN based matching approaches at various matching thresholds. The matching scores obtained by two verification schemes are fused together by a weighted score sum method. The fusion weights were selected empirically. Further, the matching threshold for FM based face verification was
Declaration of Competing Interest
The authors declare no conflicts of interest.
Acknowledgment
This work was supported by the ICT R&D By the Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government (MSIT) [Project Number : 2020-0-00113, Project Name : Development of data augmentation technology by using heterogeneous information and data fusions], and the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA contract number 2014-14071600011. The views and conclusions
Declaration of Competing Interest
The author(s) declare(s) that there is no conflict of interest regarding the publication of this paper.
References (38)
- et al.
“Unconstrained Face Verification Using Deep CNN Features,” 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY
(2016) - et al.
Facial mole detection: an approach towards face identification
Procedia Computer Science
(2015) - et al.
Facial marks: Soft biometric for face recognition
Keynote speech 1: Bag of visual soft biometrics for person identification: New trends and challenges
- et al.
Face matching and retrieval using soft biometrics
IEEE Transactions on Information Forensics and Security
(2010) - et al.
Forensic face recognition as a means to determine strength of evidence: A survey
Forensic Science Review
(2018) - et al.
Integrating faces, fingerprints, and soft biometric traits for user recognition
- et al.
Soft biometric traits for continuous user authentication
- et al.
What Else does your biometric data reveal? A survey on soft biometrics
in IEEE Transactions on Information Forensics and Security
(March 2016) - et al.
Soft biometric traits for personal recognition systems, Biometric Authentication
(2004)