Reconstructing hand-object interactions is a challenging task due to strong occlusions and complex motions. This article proposes a real-time system that uses a single depth stream to simultaneously reconstruct hand poses, object shape, and rigid/non-rigid motions. To achieve this, we first train a joint learning network to segment the hand and object in a depth image, and to predict the 3D keypoints of the hand. With most layers shared by the two tasks, computation cost is saved for the real-time performance. A hybrid dataset is constructed here to train the network with real data (to learn real-world distributions) and synthetic data (to cover variations of objects, motions, and viewpoints). Next, the depth of the two targets and the keypoints are used in a uniform optimization to reconstruct the interacting motions. Benefitting from a novel tangential contact constraint, the system not only solves the remaining ambiguities but also keeps the real-time performance. Experiments show that our system handles different hand and object shapes, various interactive motions, and moving cameras.

中文翻译：

---

基于单深度视图的手物交互实时重建

由于强烈的遮挡和复杂的运动，重建手对象交互是一项具有挑战性的任务。本文提出了一种实时系统，该系统使用单个深度流同时重建手部姿势、物体形状和刚性/非刚性运动。为此，我们首先训练一个联合学习网络来分割深度图像中的手和物体，并预测手的 3D 关键点。由于两个任务共享大多数层，因此节省了计算成本以提高实时性能。这里构建了一个混合数据集，以使用真实数据（以学习真实世界的分布）和合成数据（以涵盖对象、运动和视点的变化）来训练网络。接下来，在统一优化中使用两个目标的深度和关键点来重建相互作用的运动。受益于一种新颖的切向接触约束，该系统不仅解决了剩余的歧义，而且保持了实时性。实验表明，我们的系统可以处理不同的手和物体形状、各种交互动作和移动相机。