Generating photo-realistic training data to improve face recognition accuracy

Highlights

• Generate photo-realistic face images using a conditional GAN.

• Two latent vectors encode identity-related and non-related attributes respectively.

• Map discrete identity labels to identity features in a continuous latent space.

• Training sets with better balance between real and synthetic images outperformed.

Abstract

Face recognition has become a widely adopted biometric in forensics, security and law enforcement thanks to the high accuracy achieved by systems based on convolutional neural networks (CNNs). However, to achieve good performance, CNNs need to be trained with very large datasets which are not always available. In this paper we investigate the feasibility of using synthetic data to augment face datasets. In particular, we propose a novel generative adversarial network (GAN) that can disentangle identity-related attributes from non-identity-related attributes. This is done by training an embedding network that maps discrete identity labels to an identity latent space that follows a simple prior distribution, and training a GAN conditioned on samples from that distribution. A main novelty of our approach is the ability to generate both synthetic images of subjects in the training set and synthetic images of new subjects not in the training set, both of which we use to augment face datasets. By using recent advances in GAN training, we show that the synthetic images generated by our model are photo-realistic, and that training with datasets augmented with those images can lead to increased recognition accuracy. Experimental results show that our method is more effective when augmenting small datasets. In particular, an absolute accuracy improvement of 8.42% was achieved when augmenting a dataset of less than 60k facial images.

Introduction

Recent progress in machine learning has made possible the development of face recognition systems that can match face images as good as or better than humans. For this reason, these systems have become a valuable tool in forensics and security. However, state-of-the-art face recognition systems based on convolutional neural networks (CNNs) need to be trained with very large datasets of face images. In this work we aim to reduce the data requirement of face recognition systems by synthesising artificial face images.

Image synthesis is a widely studied topic in computer vision. In particular, face image synthesis has gained a lot of attention because of its diverse practical applications. These include facial image editing (Antipov et al., 2017, Brock et al., 2017, Choi et al., 2018, Lample et al., 2017, Larsen et al., 2016, Perarnau et al., 2016, Shu et al., 2017, Yan et al., 2016, Zhang et al., 2017), face de-identification (Brkic et al., 2017, Meden et al., 2018, Meden et al., 2017, Wu et al., 2019), data augmentation (Banerjee et al., 2017, Kortylewski et al., 2018, Masi et al., 2019, Masi et al., 2016, Mokhayeri et al., 2018, Osadchy et al., 2017, Zhao et al., 2019), face frontalisation (Hassner et al., 2015, Huang et al., 2017, Tran et al., 2019, Yim et al., 2015, Zhu et al., 2015, Zhu et al., 2013, Zhu et al., 2014) and artistic applications (e.g. video games and advertisements).

In this work, we focus on the applicability of face image synthesis for data augmentation. It is widely known that training data is one of the most important factors that affect the accuracy of deep learning models. The datasets used for training need to be large and contain sufficient variation to allow the resulting models to learn features that generalise well to unseen samples. In the case of face recognition, the datasets must contain many different subjects, as well as many different images per subject. The first requirement enables a model to learn inter-class discriminative features that can generalise to subjects not in the training set. The second requirement enables a model to learn features that are robust to intra-class variations. Even though there are several public large-scale datasets (Bansal et al., 2017, Cao et al., 2018, Guo et al., 2016, Nech and Kemelmacher-Shlizerman, 2017, Parkhi et al., 2015, Sun et al., 2014, Yi et al., 2014) that can be used to train CNN-based face recognition models, these datasets are nowhere near the size or quality of commercial datasets. For example, the largest publicly available dataset contains about 10M images of 100K different subjects (Guo et al. 2016), whereas Google’s FaceNet (Schroff et al. 2015) was trained with a private dataset containing between 100M and 200M face images of about 8M different subjects. Another issue is the presence of long-tail distributions in some publicly available datasets, i.e. datasets in which there are many subjects with very few images. Such unbalanced datasets can make the training process difficult and result in models that achieve lower accuracy than those trained with smaller but balanced datasets (Zhao et al., 2019, Zhou et al., 2015). In addition, some publicly available datasets (e.g. Guo et al. 2016) contain many mislabelled samples that can decrease face recognition accuracy if not discarded from the training set. Since collecting large-scale, good quality face datasets is a very expensive and labour-intensive task, we propose a method for generating photo-realistic face images that can be used to effectively increase the depth (number of images per subject) and width (number of subjects) of existing face datasets.

An approach that has recently gained popularity for augmenting face datasets is the use of 3D morphable models (Blanz and Vetter 2003). In this approach, new faces of existing subjects can be synthetised by fitting a 3D morphable model to existing images and modifying a variety of parameters to generate new poses and expressions (Masi et al., 2019, Masi et al., 2016, Mokhayeri et al., 2018, Zhao et al., 2019). It is also possible to generate images with other variations using this approach. For example, Mokhayeri et al. (2018) incorporated a reflectance model to generate images under different lighting conditions; and Kortylewski et al. (2018) randomly sampled 3D face shapes and colours to generate faces of new subjects. The main drawback of methods based on 3D morphable models is that the generated images often look unnatural and lack the level of detail found in real images. Another recent approach based on blending small triangular regions from different training images was proposed in Banerjee et al. (2017). Although this method seemed to produce photo-realistic faces, the authors limited their work to frontal face images. In contrast, our approach makes use of generative adversarial networks (GANs) (Goodfellow et al., 2014, Schmidhuber, 2020), which have recently been shown to produce photo-realistic in-the-wild images often indistinguishable from real images (Karras et al. 2018). Another advantage of using GANs is that they are end-to-end trainable models that do not require any domain-specific processing, as opposed to methods based on 3D modelling or face triangulation.

Many methods based on GANs have been proposed for manipulating attributes of existing face images, including age (Antipov et al., 2017, Yan et al., 2016, Zhang et al., 2017), facial expressions (Choi et al., 2018, Ding et al., 2018, Yan et al., 2016, Zhou and Shi, 2017), and other attributes such as hairstyle, glasses, makeup, facial hair, skin colour or gender (Brock et al., 2017, Choi et al., 2018, He et al., 2017, Lample et al., 2017, Larsen et al., 2016, Lu et al., 2018, Perarnau et al., 2016, Shen and Liu, 2017, Shu et al., 2017). While these methods can be used to increase the depth of a dataset, it remains unclear how to increase the width of a dataset, i.e. how to generate faces of new subjects. Our proposed GAN is able to generate faces from a latent representation $\mathit{z}$ that has two Gaussian distributed components ${\mathit{z}}_{id}$ and ${\mathit{z}}_{nid}$ encoding identity-related attributes and non-identity-related attributes respectively. In this way, face images of new subjects can be generated by fixing the identity component ${\mathit{z}}_{id}$ and varying the non-identity component ${\mathit{z}}_{nid}$. The method most closely related to ours is the semantically decomposed GAN (SD-GAN) proposed in Donahue et al. (2018). SD-GANs are trained with pairs of real images from the same subject and pairs of images generated with the same identity-related attributes but different non-identity-related attributes. A discriminator learns to reject pairs of images when either they do not look photo-realistic or when they do not appear to belong to the same subject. One of the main differences of our method with respect to SD-GANs is that it allows the generation of face images of subjects that exist in the training set. In other words, our method can increase both the width and the depth of a given face dataset. Furthermore, our proposed GAN is arguably simpler to implement than SD-GAN and easier to incorporate into other GAN architectures.

To demonstrate the efficacy of our method, we trained several CNN-based face recognition models with different combinations of real and synthetic data. In most cases, the models trained with a combination of real and synthetic data outperformed the models trained with real data alone.

Our main contributions can be summarised as:

• A novel face image synthesis method based on GANs that allows the disentangling of identity-related attributes from non-identity-related attributes.

• A data augmentation approach that uses the proposed GAN to increase the depth and width of existing face datasets.

• A experimental demonstration that the proposed data augmentation approach can be used to increase the accuracy of a face recognition algorithm trained with real images alone.

The rest of this paper is organised as follows: Section 2 provides the background needed to understand our proposed GAN. Section 3 explains each part of our proposed GAN and the loss functions used for training. Section 4 discusses our experimental results, both in terms of the quality of the synthetic images generated by our proposed GAN and the accuracy achieved by datasets augmented with the synthetic images. Finally, our conclusions are presented in Section 5.