In this work, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception and, later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable set of features, transferring the features across the domains, and classifying the objects. We show the results of our approach using a set of 15 objects, collecting 40 visual examples and five tactile examples for each object. The proposed approach achieves an accuracy of 94.7%, which is comparable with the accuracy of the monomodal case, i.e., when using visual data both as training set and test set. Moreover, it performs well compared to the human ability, which we have roughly estimated carrying out an experiment with ten participants.

中文翻译：

---

跨模态物体识别的迁移学习方法：从视觉观察到机器人触觉探索

在这项工作中，我们介绍了机器人主动探索的跨模态视觉触觉对象识别问题。有了这个术语，我们的意思是机器人通过视觉感知观察一组物体，然后它只能通过触觉探索识别这些物体，而之前没有接触过任何物体。使用机器学习术语，在我们的应用程序中，我们有一个视觉训练集和一个触觉测试集，反之亦然。为了解决这个问题，我们提出了一种由四个步骤组成的方法：找到视觉触觉共同表示，定义一组合适的特征，跨域转移特征，并对对象进行分类。我们使用一组 15 个对象展示了我们的方法的结果，为每个对象收集了 40 个视觉示例和五个触觉示例。所提出的方法实现了 94.7% 的准确度，这与单模态案例的准确度相当，即，当使用视觉数据作为训练集和测试集时。此外，与人类能力相比，它表现良好，我们粗略估计了对十名参与者进行的实验。