In this paper, a novel visual-tactile sensor is proposed; additionally, an object manipulation method for a multi-fingered robotic hand grasping an object is proposed by detecting a contact position using the visual-tactile sensor. The visual-tactile sensor is composed of a hemispheric fingertip made of soft silicone with a hollow interior and a general USB camera located inside the fingertip to detect the displacement of the many point markers embedded in the silicone. The deformation of each point marker due to a contact force is measured, and a contact position is estimated reliably through a novel method of creating virtual points to determine the point clouds. The aim is to demonstrate both the estimation performance of the new visual-tactile sensor and its usefulness in a grasping and manipulation task. By using the contact position obtained from the proposed sensor and the position of each fingertip obtained from kinematics, the position and orientation of a grasped object are estimated and controlled. The effectiveness of the method is illustrated through numerical simulation and its practical use is demonstrated through grasping and manipulating experiments.

中文翻译：

---

带有视觉触觉指尖传感器的多指机器人手进行的灵巧物体操作

本文提出了一种新型的视觉触觉传感器。另外，提出了一种通过使用视觉触觉传感器检测接触位置的，用于多指机器人手抓握物体的物体操纵方法。视觉触觉传感器由软硅胶制成的半球形指尖（内部空心）和位于指尖内侧的通用USB摄像头组成，可检测嵌入硅胶中的许多点标记的位移。测量由于接触力导致的每个点标记的变形，并通过创建虚拟点以确定点云的新颖方法可靠地估计接触位置。目的是证明新型视觉触觉传感器的估计性能及其在抓握和操纵任务中的实用性。通过使用从建议的传感器获得的接触位置和从运动学获得的每个指尖的位置，可以估算和控制所抓持物体的位置和方向。通过数值模拟说明了该方法的有效性，并通过掌握和操作实验证明了该方法的实际应用。