Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus. Unfortunately, this method fails for important cases such as highly non-rigid deformations. We first address this problem of lack of data by introducing a novel semi-supervised strategy to obtain dense inter-frame correspondences from a sparse set of annotations. This way, we obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 5,533 densely aligned frame pairs; in addition, we provide a test set along with several metrics for evaluation. Based on this corpus, we introduce a data-driven non-rigid feature matching approach, which we integrate into an optimization-based reconstruction pipeline. Here, we propose a new neural network that operates on RGB-D frames, while maintaining robustness under large non-rigid deformations and producing accurate predictions. Our approach significantly outperforms existing non-rigid reconstruction methods that do not use learned data terms, as well as learning-based approaches that only use self-supervision.

中文翻译：

---

DeepDeform：使用半监督数据学习非刚性 RGB-D 重建

将数据驱动的方法应用于非刚性 3D 重建一直很困难，我们认为这可以归因于缺乏大规模的训练语料库。不幸的是，这种方法在诸如高度非刚性变形等重要情况下失败。我们首先通过引入一种新的半监督策​​略来从一组稀疏的注释中获得密集的帧间对应关系来解决缺乏数据的问题。通过这种方式，我们获得了一个包含 400 个场景、超过 390,000 个 RGB-D 帧和 5,533 个密集对齐的帧对的大型数据集；此外，我们提供了一个测试集以及几个评估指标。基于这个语料库，我们引入了一种数据驱动的非刚性特征匹配方法，我们将其集成到基于优化的重建管道中。在这里，我们提出了一种在 RGB-D 帧上运行的新神经网络，同时在大的非刚性变形下保持鲁棒性并产生准确的预测。我们的方法明显优于现有的不使用学习数据项的非刚性重建方法，以及仅使用自我监督的基于学习的方法。