In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods.

中文翻译：

---

用于基于深度的 3D 面部姿势跟踪的可见性约束生成模型


在本文中，我们提出了一个生成框架，在具有严重遮挡和任意面部表情变化的无约束场景中，统一了基于深度的 3D 面部姿势跟踪和动态面部模型自适应。具体来说，我们引入了一种统计3D可变形模型，可以灵活地描述人脸模型表面上的点的分布，并具有高效的可切换在线自适应功能，可以逐渐捕获跟踪对象的身份，并在对象发生变化时快速构建合适的人脸模型。此外，与采用基于 ICP 的面部姿势估计的现有技术不同，为了提高对遮挡的鲁棒性，我们提出了一种光线可见性约束，该约束基于面部模型相对于输入点云的可见性来规范姿势。 Biwi 和 ICT-3DHP 数据集上的消融研究和实验结果表明，所提出的框架是有效的，并且优于完成最先进的基于深度的方法。