Objects rarely sit in isolation in everyday human environments. If we want robots to operate and perform tasks in our human environments, they must understand how the objects they manipulate will interact with structural elements of the environment for all but the simplest of tasks. As such, we would like our robots to reason about how multiple objects and environmental elements relate to one another and how those relations may change as the robot interacts with the world. We examine the problem of predicting interobject and object–environment relations between previously unseen objects and novel environments purely from partial-view point clouds. Our approach enables robots to plan and execute sequences to complete multiobject manipulation tasks defined from logical relations. This removes the burden of providing explicit, continuous object states as goals to the robot. We explore several different neural network architectures for this task. We find the best performing model to be a novel transformer-based neural network that both predicts object–environment relations and learns a latent-space dynamics function. We achieve reliable sim-to-real transfer without any fine-tuning. Our experiments show that our model understands how changes in observed environmental geometry relate to semantic relations between objects.

中文翻译：

---

使用环境感知关系分类器进行多对象操作的潜在空间规划

在人类的日常环境中，物体很少是孤立存在的。如果我们希望机器人在人类环境中操作和执行任务，它们必须了解它们操纵的物体将如何与环境的结构元素相互作用，以完成除了最简单的任务之外的所有任务。因此，我们希望我们的机器人能够推理多个物体和环境元素如何相互关联，以及当机器人与世界互动时这些关系可能如何变化。我们纯粹从部分视点云来研究预测以前未见过的物体和新环境之间的物体间关系和物体-环境关系的问题。我们的方法使机器人能够规划和执行序列，以完成根据逻辑关系定义的多对象操纵任务。这消除了向机器人提供明确、连续的对象状态作为目标的负担。我们为此任务探索了几种不同的神经网络架构。我们发现性能最好的模型是一种新型的基于变压器的神经网络，它既可以预测物体与环境的关系，又可以学习潜在空间动力学函数。我们无需任何微调即可实现可靠的模拟到真实的传输。我们的实验表明，我们的模型理解观察到的环境几何形状的变化如何与对象之间的语义关系相关。