We propose a novel robotic system that can improve its semantic perception during deployment. Contrary to the established approach of learning semantics from large datasets and deploying fixed models, we propose a framework in which semantic models are continuously updated on the robot to adapt to the deployment environments. Our system therefore tightly couples multi-sensor perception and localisation to continuously learn from self-supervised pseudo labels. We study this system in the context of a construction robot registering LiDAR scans of cluttered environments against building models. Our experiments show how the robot's semantic perception improves during deployment and how this translates into improved 3D localisation by filtering the clutter out of the LiDAR scan, even across drastically different environments. We further study the risk of catastrophic forgetting that such a continuous learning setting poses. We find memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments. On average, our system improves by 60% in segmentation and 10% in localisation compared to deployment of a fixed model, and it keeps this improvement up while adapting to further environments.

中文翻译：

---

在建筑机器人上的自我改善的语义感知

我们提出了一种新颖的机器人系统，可以在部署过程中改善其语义感知。与从大型数据集中学习语义并部署固定模型的既定方法相反，我们提出了一个框架，其中语义模型在机器人上不断更新以适应部署环境。因此，我们的系统将多传感器感知和定位紧密结合在一起，以不断从自我监督的伪标签中学习。我们在建筑机器人针对建筑模型记录杂乱环境的LiDAR扫描的情况下研究了该系统。我们的实验表明，即使在截然不同的环境中，机器人在部署过程中的语义感知也将得到改善，以及如何通过过滤掉LiDAR扫描中的杂波将其转化为改进的3D本地化。我们将进一步研究这种持续学习环境带来的灾难性遗忘风险。我们发现内存重播是减少遗忘的有效措施，并展示了即使在不同环境之间切换时，机器人系统也可以如何改善。与部署固定模型相比，我们的系统平均在细分方面提高了60％，在本地化方面提高了10％，并且可以在保持这种改进的同时适应更多的环境。