Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single “in-the-wild” image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from “in-the-wild” images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this paper, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single “in-the-wild” image. To achieve this, we capture a large dataset of facial shape and reflectance, which we have made public. Moreover, we define a fast and photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular reflectance components from a mesh and texture with baked illumination, scanned or reconstructed with a 3DMM fitting method. As we demonstrate in a series of qualitative and quantitative experiments, our method outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image, that are ready to be rendered in various applications and bridge the uncanny valley.

中文翻译：

---

AvatarMe++：使用真实感渲染感知 GAN 进行面部形状和 BRDF 推理


在过去的几年里，随着生成对抗网络 (GAN) 的出现，许多人脸分析任务已经取得了惊人的性能，其应用包括但不限于从单个“野外”进行人脸生成和 3D 人脸重建图像。然而，据我们所知，没有任何方法可以从“野外”图像生成可渲染的高分辨率 3D 人脸，这可归因于：(a) 缺乏可用于训练的数据，(b) 缺乏可以成功应用于极高分辨率数据的稳健方法。在本文中，我们介绍了第一种方法，该方法能够从单个“野外”图像重建逼真的可渲染 3D 面部几何形状和 BRDF。为了实现这一目标，我们捕获了面部形状和反射率的大型数据集，并将其公开。此外，我们定义了一种快速且逼真的可微分渲染方法，具有精确的面部皮肤漫反射和镜面反射、自遮挡和次表面散射近似。由此，我们训练了一个网络，通过烘焙照明、使用 3DMM 拟合方法扫描或重建，将面部漫反射和镜面反射分量从网格和纹理中分离出来。正如我们在一系列定性和定量实验中所证明的那样，我们的方法大大优于现有技术，并从单个低分辨率图像重建真实的 4K x 6K 分辨率 3D 面孔，并准备好在各种应用程序中渲染并架起恐怖谷的桥梁。