Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD). A significant portion of a participant's face is hidden and facial expressions are difficult to perceive. Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. In this paper, we propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware. Our approach is to track the user's face underneath the HMD utilizing a Convolutional Neural Network (CNN) and generate corresponding expressions with Generative Adversarial Networks (GAN) for producing RGBD images of the person's face. We use commodity hardware with low-cost extensions such as 3D-printed mounts and miniature cameras. Our approach learns end-to-end without manual intervention, runs in real time, and can be trained and executed on an ordinary gaming computer. We report evaluation results showing that our low-cost system does not achieve the same fidelity of research prototypes using high-end hardware and closed source software, but it is capable of creating individual facial avatars with person-specific characteristics in movements and expressions.

中文翻译：

---

揭开通信伙伴的面纱：一种低成本的人工智能解决方案，用于在基于 VR 的网真中以数字方式移除头戴式显示器

当参与者佩戴头戴式显示器 (HMD) 时，虚拟现实 (VR) 中的面对面对话是一项挑战。参与者面部的很大一部分是隐藏的，面部表情难以察觉。过去的研究表明，在实验室条件下，使用高成本硬件在 VR 中使用个人化身进行高保真人脸重建是可能的。在本文中，我们为这项任务提出了第一个低成本系统，它只使用开源、免费软件和负担得起的硬件。我们的方法是利用卷积神经网络 (CNN) 跟踪 HMD 下用户的面部，并使用生成对抗网络 (GAN) 生成相应的表情，以生成人脸的 RGBD 图像。我们使用具有低成本扩展功能的商品硬件，例如 3D 打印支架和微型相机。我们的方法无需人工干预即可端到端学习，实时运行，并且可以在普通游戏计算机上进行训练和执行。我们报告的评估结果表明，我们的低成本系统无法实现与使用高端硬件和闭源软件的研究原型相同的保真度，但它能够创建具有个人特定动作和表情特征的个人面部化身。