By overlaying virtual imagery onto the real world, mixed reality facilitates diverse applications and has drawn increasing attention. Enhancing physical in‐hand objects with a virtual appearance is a key component for many applications that require users to interact with tools such as surgery simulations. However, due to complex hand articulations and severe hand‐object occlusions, resolving occlusions in hand‐object interactions is a challenging topic. Traditional tracking‐based approaches are limited by strong ambiguities from occlusions and changing shapes, while reconstruction‐based methods show a poor capability of handling dynamic scenes. In this article, we propose a novel real‐time optimization system to resolve hand‐object occlusions by spatially reconstructing the scene with estimated hand joints and masks. To acquire accurate results, we propose a joint learning process that shares information between two models and jointly estimates hand poses and semantic segmentation. To facilitate the joint learning system and improve its accuracy under occlusions, we propose an occlusion‐aware RGB‐D hand data set that mitigates the ambiguity through precise annotations and photorealistic appearance. Evaluations show more consistent overlays compared with literature, and a user study verifies a more realistic experience.

中文翻译：

---

通过联合深度学习和模型优化解决混合现实中的手部物体遮挡问题

通过将虚拟图像叠加到现实世界上，混合现实促进了多种应用，并引起了越来越多的关注。对于需要用户与手术模拟等工具进行交互的许多应用程序来说，增强具有虚拟外观的实际手头对象是一个关键组件。然而，由于复杂的手部关节和严重的手部对象遮挡，解决手部对象交互中的遮挡是一个具有挑战性的话题。传统的基于跟踪的方法受到遮挡和变化形状的强烈模糊性的限制，而基于重建的方法处理动态场景的能力很差。在本文中，我们提出了一种新颖的实时优化系统，通过使用估计的手部关节和蒙版在空间上重建场景来解决手部对象遮挡问题。为了获得准确的结果，我们提出了一个联合学习过程，在两个模型之间共享信息并联合估计手部姿势和语义分割。为了促进联合学习系统并提高其在遮挡下的准确性，我们提出了一个遮挡感知 RGB-D 手部数据集，它通过精确的注释和逼真的外观来减轻歧义。与文献相比，评估显示更一致的叠加，用户研究验证了更真实的体验。