In this paper, we proposed a semi-supervised common representation learning method with GAN-based Asymmetric Transfer Network (GATN) for cross modality retrieval. GATN utilizes the asymmetric pipeline to guarantee the semantic consistency and adopt (Generative Adversarial Network) GAN to fit the distributions of different modalities. Specifically, the common representation learning across modalities includes two stages: (1) the first stage, GATN trains source mapping network to learn the semantic representation of text modality by supervised method; and (2) the second stage, GAN-based unsupervised modality transfer method is proposed to guide the training of target mapping network, which includes generative network (target mapping network) and discriminative network. Experimental results on three widely-used benchmarks show that GATN have achieved better performance comparing with several existing state-of-the-art methods.

在本文中，我们提出了一种基于GAN的非对称传输网络（GATN）的半监督通用表示学习方法，用于跨模态检索。GATN利用非对称流水线来保证语义一致性，并采用GAN（通用对抗网络）来适应不同形式的分布。具体来说，跨模式的通用表示学习包括两个阶段：（1）第一阶段，GATN训练源映射网络通过监督方法学习文本模式的语义表示；（2）第二阶段，提出了基于GAN的无监督模态传递方法，指导目标映射网络的训练，包括生成网络（目标映射网络）和判别网络。