Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are hand-crafted, a scaleable way to build simulators is to use machine learning to learn how the environment behaves in response to an action, directly from data. In this work, we aim to learn to simulate a dynamic environment directly in pixel-space, by watching unannotated sequences of frames and their associated action pairs. We introduce a novel high-quality neural simulator referred to as DriveGAN that achieves controllability by disentangling different components without supervision. In addition to steering controls, it also includes controls for sampling features of a scene, such as the weather as well as the location of non-player objects. Since DriveGAN is a fully differentiable simulator, it further allows for re-simulation of a given video sequence, offering an agent to drive through a recorded scene again, possibly taking different actions. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data. We showcase that our approach greatly surpasses the performance of previous data-driven simulators, and allows for new features not explored before.

中文翻译：

---

DriveGAN：迈向可控的高质量神经仿真

现实的模拟器对于培训和验证机器人系统至关重要。尽管大多数现代模拟器都是手工制作的，但构建模拟器的一种可扩展的方法是使用机器学习直接从数据中了解环境如何响应动作。在这项工作中，我们旨在通过观察未注释的帧序列及其关联的动作对，学习直接在像素空间中模拟动态环境。我们介绍了一种称为DriveGAN的新型高质量神经仿真器，该仿真器通过在不需要监督的情况下解开不同的组件来实现可控性。除了操纵控件外，它还包括用于采样场景特征（例如天气以及非玩家对象的位置）的控件。由于DriveGAN是完全可区分的模拟器，它还允许重新模拟给定的视频序列，从而提供代理以再次驱动通过记录的场景，并可能采取不同的动作。我们在多个数据集上对DriveGAN进行训练，包括160个小时的实际驾驶数据。我们展示了我们的方法大大超越了以前的数据驱动模拟器的性能，并允许使用以前未曾探索过的新功能。