As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.

中文翻译：

---

机器人代理可重复和可访问评估的集成基准和设计

随着机器人技术的成熟和复杂性的增加，机器人自主研究的可重复性比以往任何时候都更加必要。与其他科学相比，对自主性进行基准测试存在特定挑战，例如软件堆栈的复杂性、硬件的可变性以及对数据驱动技术的依赖等。在本文中，我们描述了一个可重复的机器人研究的新概念，它集成了开发和基准测试，以便从研究/开发过程的开始就“通过设计”获得可重复性。我们首先提供实现这一目标的总体概念目标，然后是我们构建的具体实例：DUCKIENet。此设置的核心组件之一是 Duckietown Autolab，一种可远程访问的标准化设置，其本身的成本也相对较低且可重复。在评估代理时，仔细定义接口允许用户使用模拟、日志或远程自动化硬件设置在本地或远程评估之间进行选择。我们通过分析使用基础设施进行的实验的可重复性来验证系统，并表明不同机器人硬件和不同远程实验室之间的差异很小。