Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

中文翻译：

---

基于 3D 视觉的机器人操纵的综合研究


机器人操纵，例如拾放操纵，广泛应用于工业机器人的智能制造、水下机器人的海洋工程、服务机器人，甚至医疗机器人的医疗保健。大多数传统的机器人操纵采用具有平面假设的2D视觉系统，并且只能相应地生成3-DOF（自由度）位姿。为了模仿人类智能并赋予机器人更灵活的工作能力，基于3D视觉的机器人操控得到了研究。然而，这项任务在开放世界中仍然具有挑战性，特别是对于杂乱背景中的一般对象识别和遮挡姿态估计以及类人灵活操作。在本文中，我们对用于机器人操纵的 3D 视觉的最新进展进行了全面分析，包括 3D 数据采集和表示、机器人视觉校准、3D 物体检测/识别、6-DOF 位姿估计、抓取估计和运动规划。然后，我们提供一些公共数据集、评估标准、比较和挑战。最后，给出了机器人操纵的相关应用领域，并研究了一些未来的方向和开放性问题。