This paper presents a method for riggable 3D face reconstruction from monocular images, which jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. To achieve this goal, we design an end-to-end trainable network embedded with a differentiable in-network optimization. The network first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization. By estimating a personalized face rig, our method goes beyond static reconstructions and enables downstream applications such as video retargeting. In-network optimization explicitly enforces constraints derived from the first principles, thus introduces additional priors than regression-based methods. Finally, data-driven priors from deep learning are utilized to constrain the ill-posed monocular setting and ease the optimization difficulty. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability, and supports standard face rig applications.

中文翻译：

---

通过网络内优化进行可操纵的3D人脸重建

本文提出了一种用于从单眼图像进行可绑定3D脸部重构的方法，该方法可以联合估计个性化脸部绑定和每个图像参数，包括表情，姿势和照明。为了实现这个目标，我们设计了一个嵌入了端到端的可训练网络，并进行了可区分的网络内优化。网络首先使用神经解码器将面部装备参数化为紧凑的潜在代码，然后通过可学习的优化来估计潜在代码以及每个图像参数。通过估计个性化的面部装备，我们的方法不仅可以进行静态重建，还可以实现下游应用，例如视频重定向。网络内优化显式地强加了从第一个原理派生的约束，因此引入了比基于回归的方法更多的先验条件。最后，深度学习的数据驱动先验被用来约束不适定的单眼环境并减轻优化难度。实验表明，该方法达到了SOTA重构精度，合理的鲁棒性和泛化能力，并支持标准的面部钻机应用。