We demonstrate model-based, visual robot manipulation of linear deformable objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is in contrast to dynamics learned in pixel space or latent space, where generalization to visual differences are not guaranteed. Challenges in taking the state-space approach are the estimation of the high-dimensional state of a deformable object from raw images, where annotations are very expensive on real data, and finding a dynamics model that is both accurate, generalizable, and efficient to compute. We are the first to demonstrate self-supervised training of rope state estimation on real images, without requiring expensive annotations. This is achieved by our novel self-supervising learning objective, which is generalizable across a wide range of visual appearances. With estimated rope states, we train a fast and differentiable neural network dynamics model that encodes the physics of mass-spring systems. Our method has a higher accuracy in predicting future states compared to models that do not involve explicit state estimation and do not use any physics prior, while only using 3\% of training data. We also show that our approach achieves more efficient manipulation, both in simulation and on a real robot, when used within a model predictive controller.

中文翻译：

---

用于操纵可变形线性对象的状态估计的自监督学习

我们展示了对线性可变形物体的基于模型的视觉机器人操作。我们的方法基于机器人旨在控制的物理系统的状态空间表示。这种选择具有多种优势，包括易于在动力学模型和感知模型中结合物理先验，以及易于规划操作动作。此外，物理状态可以自然地表示不同外观的对象实例。因此，状态空间中的动态可以在一种设置中学习，并直接用于其他视觉上不同的设置。这与在像素空间或潜在空间中学习的动态形成对比，其中不能保证对视觉差异的泛化。采用状态空间方法的挑战是从原始图像中估计可变形物体的高维状态，其中对真实数据的注释非常昂贵，并找到一个既准确、可推广且计算效率高的动力学模型. 我们是第一个在真实图像上展示绳索状态估计的自监督训练的人，不需要昂贵的注释。这是通过我们新颖的自我监督学习目标实现的，该目标可在广泛的视觉外观中推广。通过估计的绳索状态，我们训练了一个快速且可微的神经网络动力学模型，该模型对质量弹簧系统的物理学进行编码。与不涉及显式状态估计且不使用任何先验物理的模型相比，我们的方法在预测未来状态方面具有更高的准确性，而只使用 3\% 的训练数据。我们还表明，当在模型预测控制器中使用时，我们的方法在模拟和真实机器人上都实现了更有效的操作。