Near-infrared to visible (NIR-VIS) face recognition is the most common case in heterogeneous face recognition, which aims to match a pair of face images captured from two different modalities. Existing deep learning based methods have made remarkable progress in NIR-VIS face recognition, while it encounters certain newly-emerged difficulties during the pandemic of COVID-19, since people are supposed to wear facial masks to cut off the spread of the virus. We define this task as NIR-VIS masked face recognition, and find it problematic with the masked face in the NIR probe image. First, the lack of masked face data is a challenging issue for the network training. Second, most of the facial parts (cheeks, mouth, nose etc.) are fully occluded by the mask, which leads to a large amount of loss of information. Third, the domain gap still exists in the remaining facial parts. In such scenario, the existing methods suffer from significant performance degradation caused by the above issues. In this paper, we aim to address the challenge of NIR-VIS masked face recognition from the perspectives of training data and training method. Specifically, we propose a novel heterogeneous training method to maximize the mutual information shared by the face representation of two domains with the help of semi-siamese networks. In addition, a 3D face reconstruction based approach is employed to synthesize masked face from the existing NIR image. Resorting to these practices, our solution provides the domain-invariant face representation which is also robust to the mask occlusion. Extensive experiments on three NIR-VIS face datasets demonstrate the effectiveness and cross-dataset-generalization capacity of our method.

中文翻译：

---

迈向 NIR-VIS 蒙面人脸识别


近红外到可见光（NIR-VIS）人脸识别是异构人脸识别中最常见的情况，其目的是匹配从两种不同模式捕获的一对人脸图像。现有的基于深度学习的方法在 NIR-VIS 人脸识别方面取得了显着进展，但在 COVID-19 大流行期间遇到了一些新出现的困难，因为人们应该戴口罩来切断病毒的传播。我们将此任务定义为 NIR-VIS 蒙面人脸识别，并发现 NIR 探测图像中的蒙面人脸存在问题。首先，蒙面人脸数据的缺乏对于网络训练来说是一个具有挑战性的问题。其次，大部分面部部位（脸颊、嘴巴、鼻子等）被面具完全遮挡，导致大量信息丢失。第三，其余面部部位仍然存在域间隙。在这种情况下，现有方法会因上述问题而导致性能显着下降。在本文中，我们旨在从训练数据和训练方法的角度解决NIR-VIS蒙面人脸识别的挑战。具体来说，我们提出了一种新颖的异构训练方法，以借助半暹罗网络最大化两个域的人脸表示共享的互信息。此外，还采用基于 3D 人脸重建的方法从现有 NIR 图像合成蒙面人脸。借助这些实践，我们的解决方案提供了域不变的面部表示，该表示对于掩模遮挡也具有鲁棒性。对三个 NIR-VIS 人脸数据集的广泛实验证明了我们方法的有效性和跨数据集泛化能力。