Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

中文翻译：

---

基于手势估计和主动视觉的灵巧手臂遥操作系统


利用计算机视觉创新的无标记视觉远程操作具有允许多指机器人手进行自然且非侵入性手指运动的优势。然而，由于手指的自遮挡，当前的姿态估计方法仍然面临不准确的问题。在这里，我们开发了一种新颖的基于视觉的手臂遥控系统，该系统可以从最佳视角和合适的距离捕捉人手。该远程操作系统由端到端手势回归网络和受控主动视觉系统组成。端到端姿态回归网络（Transteleop）结合辅助重建损失函数，通过低成本深度相机捕获人手，并基于图像到图像翻译方法预测机器人的关节命令。为了获得人手的最佳观察，主动视觉系统由机器人手臂在本地实现，以确保所提出的神经网络的高精度。人臂运动在相对控制下同时映射到从属机器人臂。定量网络评估和各种复杂的操纵任务，例如塔楼、浇注和多桌杯子堆叠，证明了所提出的远程操作系统的实用性和稳定性。