To get clear street-view and photo-realistic simulation in autonomous driving, we present an automatic video inpainting algorithm that can remove traffic agents from videos and synthesize missing regions with the guidance of depth/point cloud. By building a dense 3D map from stitched point clouds, frames within a video are geometrically correlated via this common 3D map. In order to fill a target inpainting area in a frame, it is straightforward to transform pixels from other frames into the current one with correct occlusion. Furthermore, we are able to fuse multiple videos through 3D point cloud registration, making it possible to inpaint a target video with multiple source videos. The motivation is to solve the long-time occlusion problem where an occluded area has never been visible in the entire video. To our knowledge, we are the first to fuse multiple videos for video inpainting. To verify the effectiveness of our approach, we build a large inpainting dataset in the real urban road environment with synchronized images and Lidar data including many challenge scenes, e.g., long time occlusion. The experimental results show that the proposed approach outperforms the state-of-the-art approaches for all the criteria, especially the RMSE (Root Mean Squared Error) has been reduced by about 13%.

中文翻译：

---

DVI：用于自动驾驶的深度引导视频修复

为了在自动驾驶中获得清晰的街景和逼真的模拟，我们提出了一种自动视频修复算法，该算法可以从视频中去除交通代理并在深度/点云的指导下合成缺失区域。通过从拼接点云构建密集的 3D 地图，视频中的帧通过这个常见的 3D 地图在几何上相关。为了填充帧中的目标修复区域，可以直接将其他帧的像素转换为具有正确遮挡的当前像素。此外，我们能够通过 3D 点云配准融合多个视频，从而可以使用多个源视频修复目标视频。其动机是解决长时间遮挡问题，即整个视频中从未出现过遮挡区域。据我们所知，我们是第一个融合多个视频进行视频修复的公司。为了验证我们方法的有效性，我们在真实的城市道路环境中构建了一个大型修复数据集，其中包含同步图像和激光雷达数据，包括许多挑战场景，例如长时间遮挡。实验结果表明，所提出的方法在所有标准上都优于最先进的方法，尤其是 RMSE（均方根误差）降低了约 13%。