In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates not only joint angles but also depth images of paired robot hand poses through an image-to-image translation process. A keypoint-based reconstruction loss explores the resemblance in appearance and anatomy between human and robotic hands and enriches the local features of reconstructed images. A wearable camera holder enables simultaneous hand-arm control and facilitates the mobility of the whole teleoperation system. Network evaluation results on a test dataset and a variety of complex manipulation tasks that go beyond simple pick-and-place operations show the efficiency and stability of our multimodal teleoperation system.

中文翻译：

---

基于视觉和IMU的移动机器人手臂遥操作系统

在本文中，我们提出了一种多模式移动远程操作系统，该系统由一种新颖的基于视觉的手姿势回归网络（Transteleop）和一种基于 IMU 的手臂跟踪方法组成。Transteleop 通过低成本深度相机观察人手，不仅生成关节角度，还通过图像到图像转换过程生成配对机器人手姿势的深度图像。基于关键点的重建损失探索了人类和机器人手在外观和解剖结构上的相似性，并丰富了重建图像的局部特征。可穿戴式相机支架可实现同时进行手臂控制，并促进整个远程操作系统的移动性。