当前位置: X-MOL 学术IEEE Trans. Multimedia › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Query Reconstruction Network for Referring Expression Image Segmentation
IEEE Transactions on Multimedia ( IF 7.3 ) Pub Date : 2020-04-30 , DOI: 10.1109/tmm.2020.2991504
Hengcan Shi , Hongliang Li , Qingbo Wu , King Ngi Ngan

Referring expression image segmentation aims at segmenting out the object described by a natural language query. Due to the diversity of visual content and language descriptions, it is very challenging to accurately model the correspondence between the vision and language, which inevitably produces some undesired segmentation objects from the queries. In this paper, we propose a query reconstruction network (QRN) to build more consistent corresponding relations between the language queries and object segmentation results. QRN not only generates segmentations from the queries and images but also reversely reconstructs the queries from the segmentations and the images. Through query reconstruction, QRN can confirm the vision-language consistency between the segmentations and queries. In the inference stage, for inconsistent segmentations and queries, we propose an iterative segmentation correction (ISC) method to correct them. ISC takes the difference between the reconstructed and input queries as a loss to optimize the proposed QRN. Then, the proposed QRN can generate new segmentations and queries. By iterative optimization, the segmentations can be gradually corrected. Extensive experiments on four referring expression image segmentation databases demonstrate the effectiveness of the proposed method.

中文翻译:

用于参考表达图像分割的查询重建网络

引用表达图像分割旨在分割出自然语言查询所描述的对象。由于视觉内容和语言描述的多样性,因此准确地建模视觉和语言之间的对应关系非常困难,这不可避免地会从查询中产生一些不需要的分割对象。在本文中,我们提出了一种查询重建网络(QRN),以在语言查询和对象分割结果之间建立更一致的对应关系。QRN不仅从查询和图像生成分割,而且从分割和图像反向重建查询。通过查询重建,QRN可以确认细分和查询之间的视觉语言一致性。在推断阶段,对于不一致的细分和查询,我们提出了一种迭代分段校正(ISC)方法来对其进行校正。ISC将重构查询和输入查询之间的差异作为损失来优化拟议的QRN。然后,提出的QRN可以生成新的细分和查询。通过迭代优化,可以逐步校正细分。在四个参考表达图像分割数据库上的大量实验证明了该方法的有效性。
更新日期:2020-04-30
down
wechat
bug