We propose a method at the intersection of Computer Vision and Computer Graphics fields, which automatically generates RGBD images using neural networks, based on previously seen and synchronized video, depth and pose signals. Since the models must be able to reconstruct both texture (RGB) and structure (Depth), it creates an implicit representation of the scene, as opposed to explicit ones, such as meshes or point clouds. The process can be thought of as neural rendering, where we obtain a function f : Pose -> RGBD, which we can use to navigate through the generated scene, similarly to graphics simulations. We introduce two new datasets, one based on synthetic data with full ground truth information, while the other one being recorded from a drone flight in an university campus, using only video and GPS signals. Finally, we propose a fully unsupervised method of generating datasets from videos alone, in order to train the Pose2RGBD networks. Code and datasets are available at:: https://gitlab.com/mihaicristianpirvu/pose2rgbd.

中文翻译：

---

姿势2RGBD。从绝对位置生成深度和 RGB 图像

我们在计算机视觉和计算机图形领域的交叉点提出了一种方法，该方法使用神经网络基于先前看到和同步的视频、深度和姿势信号自动生成 RGBD 图像。由于模型必须能够重建纹理 (RGB) 和结构 (深度)，因此它创建了场景的隐式表示，而不是显式表示，例如网格或点云。这个过程可以被认为是神经渲染，我们得到一个函数 f : Pose -> RGBD，我们可以用它来导航生成的场景，类似于图形模拟。我们引入了两个新数据集，一个基于具有完整地面实况信息的合成数据，而另一个是从大学校园的无人机飞行中记录的，仅使用视频和 GPS 信号。最后，我们提出了一种完全无监督的方法，仅从视频生成数据集，以训练 Pose2RGBD 网络。代码和数据集位于：：https://gitlab.com/mihaicristianpirvu/pose2rgbd。