State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

中文翻译：

---

InpaintFusion：用于3D场景的增量RGB-D修复。

用于减少现实的最新方法将像素信息从关键帧传播到后续帧以进行实时修复。但是，如果场景几何形状不够平坦，则这些方法会产生伪影。在本文中，我们介绍了InpaintFusion，这是一种新的实时方法，通过在修复过程中同时考虑颜色和深度信息，可以将修复扩展到非平面场景。我们使用RGB-D传感器同时进行定位和地图绘制，以便除了RGB图像外，还可以跟踪摄像机并获得冲浪图。我们在颜色和几何外观的成本函数中使用RGB-D信息，以导出用于同时修补颜色和深度的全局优化。通过深度融合将修补的深度合并到全局地图中。对于最终渲染，我们将地图模型投影到图像空间中，在其中可以将其用于效果，例如对其他隐藏结构进行重新照明和立体渲染。我们通过将其与使用平面几何代理的方法进行修复的结果进行比较，证明了我们方法的功能。