Traditional methods for image-based 3D face reconstruction and facial motion retargeting fit a 3D morphable model (3DMM) to the face, which has limited modeling capacity and fail to generalize well to in-the-wild data. Use of deformation transfer or multilinear tensor as a personalized 3DMM for blendshape interpolation does not address the fact that facial expressions result in different local and global skin deformations in different persons. Moreover, existing methods learn a single albedo per user which is not enough to capture the expression-specific skin reflectance variations. We propose an end-to-end framework that jointly learns a personalized face model per user and per-frame facial motion parameters from a large corpus of in-the-wild videos of user expressions. Specifically, we learn user-specific expression blendshapes and dynamic (expression-specific) albedo maps by predicting personalized corrections on top of a 3DMM prior. We introduce novel constraints to ensure that the corrected blendshapes retain their semantic meanings and the reconstructed geometry is disentangled from the albedo. Experimental results show that our personalization accurately captures fine-grained facial dynamics in a wide range of conditions and efficiently decouples the learned face model from facial motion, resulting in more accurate face reconstruction and facial motion retargeting compared to state-of-the-art methods.

中文翻译：

---

用于改进人脸重建和运动重定向的个性化人脸建模

基于图像的 3D 面部重建和面部运动重定向的传统方法将 3D 可变形模型 (3DMM) 拟合到面部，其建模能力有限且无法很好地泛化到野外数据。使用变形传递或多线性张量作为个性化 3DMM 进行混合形状插值并没有解决面部表情导致不同人的局部和全局皮肤变形的事实。此外，现有方法为每个用户学习单个反照率，这不足以捕获特定于表情的皮肤反射率变化。我们提出了一个端到端框架，该框架从大量的用户表情的野外视频语料库中联合学习每个用户的个性化面部模型和每帧面部运动参数。具体来说，我们通过在 3DMM 之前预测个性化校正来学习特定于用户的表情混合形状和动态（特定于表情的）反照率图。我们引入了新的约束以确保校正后的混合形状保留其语义，并且重建的几何图形与反照率分开。实验结果表明，与最先进的方法相比，我们的个性化准确地捕获了各种条件下的细粒度面部动态，并有效地将学习到的面部模型与面部运动分离，从而实现了更准确的面部重建和面部运动重定向. 我们引入了新的约束以确保校正后的混合形状保留其语义，并且重建的几何图形与反照率分开。实验结果表明，与最先进的方法相比，我们的个性化准确地捕获了各种条件下的细粒度面部动态，并有效地将学习到的面部模型与面部运动分离，从而实现了更准确的面部重建和面部运动重定向. 我们引入了新的约束以确保校正后的混合形状保留其语义，并且重建的几何图形与反照率分开。实验结果表明，与最先进的方法相比，我们的个性化准确地捕获了各种条件下的细粒度面部动态，并有效地将学习到的面部模型与面部运动分离，从而实现了更准确的面部重建和面部运动重定向.