当前位置: X-MOL 学术IEEE Trans. Autom. Sci. Eng. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Cross-Modal Material Perception for Novel Objects: A Deep Adversarial Learning Method
IEEE Transactions on Automation Science and Engineering ( IF 5.6 ) Pub Date : 2019-10-09 , DOI: 10.1109/tase.2019.2941230
Wendong Zheng , Huaping Liu , Bowen Wang , Fuchun Sun

To more actively perform fine manipulation tasks in the real world, intelligent robots should be able to understand and communicate the physical attributes of the material during interaction with an object. Tactile and vision are two important sensing modalities in robotic perception system. In this article, we propose a cross-modal material perception framework for recognizing novel objects. Concretely, it first adopts an object-agnostic method to associate information from tactile and visual modalities. It then recognizes a novel object by using its tactile signal to retrieve perceptually similar surface material images through the learned cross-modal correlation. This problem exhibits a challenge because data from visual and tactile modalities are highly heterogeneous and weakly paired. Moreover, the framework should not only consider cross-modal pairwise relevance but also be discriminative and generalized for unseen objects. To this end, we propose a weakly paired cross-modal adversarial learning (WCMAL) model for the visual–tactile cross-modal retrieval, which combines the advantages of deep learning and adversarial learning. In particular, the model fully considers the weak pairing problem between the two modalities. Finally, we conduct verification experiments on a publicly available data set. The results demonstrate the effectiveness of the proposed method. Note to Practitioners— Since cross-modal perception can improve the active operation of automation systems, it is invaluable for industrial intelligence, particularly when only one sensing modality cannot be used or suitable in some applications. In this article, we provide a framework of cross-modal material perception for object recognition using the idea of the cross-modal retrieval. Concretely, we use relevant tactile data of an unknown object to retrieve perceptually similar surface images, which are used to evaluate its material properties. Different from that previous works using tactile information as a complement or alternative to visual information to recognize specific objects, our proposed framework is able to estimate and infer material properties of both seen and unseen objects, which can enhance manipulation systems intelligence and improve the quality of the interaction. In our future works, more modality information will be incorporated to further enhance the cross-modal material perception.

中文翻译:

新对象的跨模态材料感知:一种深度对抗学习方法

为了更主动地执行现实世界中的精细操作任务,智能机器人应该能够在与对象交互期间理解并传达材料的物理属性。触觉和视觉是机器人感知系统中的两个重要传感方式。在本文中,我们提出了一种用于识别新颖物体的跨模式物质感知框架。具体而言,它首先采用一种与对象无关的方法来关联来自触觉和视觉方式的信息。然后,它通过使用其触觉信号,通过学习到的交叉模态相关性,检索感知相似的表面物质图像,从而识别出一个新颖的物体。由于来自视觉和触觉模态的数据高度异质且配对较弱,因此该问题面临挑战。此外,该框架不仅应考虑跨模式的成对相关性,而且应对未见对象进行区分和概括。为此,我们提出了一种弱配对的跨模态对抗学习(WCMAL)模型,用于视觉-触觉跨模态检索,该模型结合了深度学习和对抗学习的优势。特别是,该模型充分考虑了两种模态之间的弱配对问题。最后,我们对公开可用的数据集进行验证实验。结果证明了该方法的有效性。结合了深度学习和对抗学习的优势。特别是,该模型充分考虑了两种模态之间的弱配对问题。最后,我们对公开可用的数据集进行验证实验。结果证明了该方法的有效性。结合了深度学习和对抗学习的优势。特别是,该模型充分考虑了两种模态之间的弱配对问题。最后,我们对公开可用的数据集进行验证实验。结果证明了该方法的有效性。执业者注意事项— 由于跨模态感知可以改善自动化系统的主动运行,因此对于工业智能而言,这是无价的,尤其是在某些应用中无法使用或不适合使用一种传感方式时。在本文中,我们使用跨模式检索的思想为对象识别提供了跨模式材料感知的框架。具体而言,我们使用未知物体的相关触觉数据来检索在感知上相似的表面图像,这些图像用于评估其材料属性。与以前使用触觉信息作为视觉信息的补充或替代来识别特定对象的先前工作不同,我们提出的框架能够估算和推断可见和不可见对象的物质特性,可以增强操纵系统的智能性并提高交互质量。在我们未来的工作中,将包含更多的情态信息,以进一步增强对跨模态材料的感知。
更新日期:2020-04-22
down
wechat
bug