Post-comparison mitigation of demographic bias in face recognition using fair score normalization

Highlights

• Enhance face verification performance for demographics effected by decision bias.

• Operates on comparison score-level to enable using existing face recognition models.

• Based on the notation of individual fairness to treat similar face groups similarly.

• Effectiveness proved on three databases and two face embeddings.

• Unsupervised enhancement of the overall and intra-class recognition performance.

Abstract

Current face recognition systems achieve high progress on several benchmark tests. Despite this progress, recent works showed that these systems are strongly biased against demographic sub-groups. Consequently, an easily integrable solution is needed to reduce the discriminatory effect of these biased systems. Previous work mainly focused on learning less biased face representations, which comes at the cost of a strongly degraded overall recognition performance. In this work, we propose a novel unsupervised fair score normalization approach that is specifically designed to reduce the effect of bias in face recognition and subsequently lead to a significant overall performance boost. Our hypothesis is built on the notation of individual fairness by designing a normalization approach that leads to treating “similar” individuals “similarly”. Experiments were conducted on three publicly available datasets captured under controlled and in-the-wild circumstances. Results demonstrate that our solution reduces demographic biases, e.g. by up to 82.7% in the case when gender is considered. Moreover, it mitigates the bias more consistently than existing works. In contrast to previous works, our fair normalization approach enhances the overall performance by up to 53.2% at false match rate of ${10}^{-3}$ and up to 82.9% at a false match rate of ${10}^{-5}$. Additionally, it is easily integrable into existing recognition systems and not limited to face biometrics.

Introduction

Large-scale face recognition systems are spreading worldwide and are increasingly involved in critical decision-making processes, such as in forensics and law enforcement. Consequently, these systems also have a growing effect on everybody’s daily life. However, current biometric solutions are mainly optimized for maximum recognition accuracy [1] and are heavily biased for certain demographic groups [2], [3], [4], [5], [6], [7]. This means that, for example, specific demographic groups can be falsely identified as black-listed individuals more frequently than other groups. Consequently, there is an increased need that guarantees fairness for biometric solutions [6], [8], [9] to prevent discriminatory decisions.

From a political perspective, there are several regulations to guarantee fairness. Article 7 of the Universal Declaration on Human Rights and Article 14 of the European Convention of Human Rights ensure people the right to non-discrimination. Also the General Data Protection Regulation (GDPR) [10] aims at preventing discriminatory effects (article 71). In spite of these political efforts, several works [2], [3], [4], [5], [6], [7] showed that commercial [6], as well as open-source [2] face recognition systems, are strongly biased towards different demographic groups. Consequently, there is an increased need for fair and unbiased biometric solutions [2], [7].

Recent works mainly focused on learning less-biased face representations [11], [12], [13], [14], [15], [16], [17] for specific demographics. However, this requires computationally expensive template-replacement of the whole database if the recognition system is updated. Moreover, the bias-mitigation often comes at the cost of a strong decrease in recognition performance.

In this work, we propose a novel and unsupervised fair score normalization bias mitigation approach that is easily-integrable. Unlike previous work, increasing fairness also leads to an improved performance of the system in total. Our theoretical motivation is based on the notation of individual fairness [18], resulting in a solution that treats similar individuals similarly and thus, more fairly. The proposed approach clusters samples in the embedding space such that similar identities are categorized without the need for pre-defined demographic classes. For each cluster, an optimal local threshold is computed and used to develop a score normalization approach that ensures a more individual, unbiased, and fair treatment. The experiments are conducted on three publicly available datasets captured under controlled and in-the-wild conditions and on two face embeddings. To justify the concept of our fair normalization approach, we provide a visual illustration that demonstrates (a) the suitability of the notation of individual fairness for face recognition and (b) the need for more individualized treatment of face recognition systems. The results show a higher consistency and efficiency of our unsupervised normalization approach compared to related works. It efficiently mitigates demographic-bias by up to 82.7% while consistently enhancing the total recognition performance by up to 82.9%. The source code for this work is available at the following link.¹