Does progress in simulation translate to progress in robotics? Specifically, if method A outperforms method B in simulation, how likely is the trend to hold in reality on a robot? We examine this question for embodied (PointGoal) navigation, developing engineering tools and a research paradigm for evaluating a simulator by its sim2real predictivity, revealing surprising findings about prior work. First, we develop Habitat-PyRobot Bridge (HaPy), a library for seamless execution of identical code on a simulated agent and a physical robot. Habitat-to-Locobot transfer with HaPy involves just one line change in config, essentially treating reality as just another simulator! Second, we investigate sim2real predictivity of Habitat-Sim for PointGoal navigation. We 3D-scan a physical lab space to create a virtualized replica, and run parallel tests of 9 different models in reality and simulation. We present a new metric called Sim-vs-Real Correlation Coefficient (SRCC) to quantify sim2real predictivity. Our analysis reveals several important findings. We find that SRCC for Habitat as used for the CVPR19 challenge is low (0.18 for the success metric), which suggests that performance improvements for this simulator-based challenge would not transfer well to a physical robot. We find that this gap is largely due to AI agents learning to 'cheat' by exploiting simulator imperfections: specifically, the way Habitat allows for 'sliding' along walls on collision. Essentially, the virtual robot is capable of cutting corners, leading to unrealistic shortcuts through non-navigable spaces. Naturally, such exploits do not work in the real world where the robot stops on contact with walls. Our experiments show that it is possible to optimize simulation parameters to enable robots trained in imperfect simulators to generalize learned skills to reality (e.g. improving $SRCC_{Succ}$ from 0.18 to 0.844).

中文翻译：

---

Sim2Real 预测性：模拟中的评估是否可以预测真实世界的性能

模拟的进步会转化为机器人技术的进步吗？具体来说，如果方法 A 在模拟中胜过方法 B，那么在机器人上实际保持这种趋势的可能性有多大？我们针对具体化 (PointGoal) 导航、开发工程工具和通过模拟器的 sim2real 预测性评估模拟器的研究范式研究了这个问题，揭示了有关先前工作的惊人发现。首先，我们开发了 Habitat-PyRobot Bridge (HaPy)，这是一个用于在模拟代理和物理机器人上无缝执行相同代码的库。HaPy 的 Habitat-to-Locobot 传输只涉及配置中的一行更改，基本上将现实视为另一个模拟器！其次，我们研究了用于 PointGoal 导航的 Habitat-Sim 的 sim2real 预测性。我们对物理实验室空间进行 3D 扫描以创建虚拟化副本，并在现实和模拟中运行 9 个不同模型的并行测试。我们提出了一个称为 Sim-vs-Real 相关系数 (SRCC) 的新指标来量化 sim2real 预测性。我们的分析揭示了几个重要的发现。我们发现用于 CVPR19 挑战的 Habitat SRCC 很低（成功指标为 0.18），这表明这种基于模拟器的挑战的性能改进不会很好地转移到物理机器人上。我们发现这种差距主要是由于 AI 代理通过利用模拟器缺陷来学习“作弊”：特别是 Habitat 允许在碰撞时沿着墙壁“滑动”的方式。从本质上讲，虚拟机器人能够偷工减料，从而通过不可导航的空间导致不切实际的捷径。自然，这种漏洞在机器人与墙壁接触时停止的现实世界中不起作用。我们的实验表明，可以优化模拟参数，使在不完美模拟器中训练的机器人能够将学到的技能推广到现实中（例如将 $SRCC_{Succ}$ 从 0.18 提高到 0.844）。