Abstract Cross-modal image generation is an important aspect of the multi-modal learning. Existing methods usually use the semantic feature to reduce the modality gap. Although these methods have achieved notable progress, there are still some limitations: (1) they usually use single modality information to learn the semantic feature; (2) they require the training data to be paired. To overcome these problems, we propose a novel semi-supervised cross-modal image generation method, which consists of two semantic networks and one image generation network. Specifically, in the semantic networks, we use image modality to assist non-image modality for semantic feature learning by using a deep mutual learning strategy. In the image generation network, we introduce an additional discriminator to reduce the image reconstruction loss. By leveraging large amounts of unpaired data, our method can be trained in a semi-supervised manner. Extensive experiments demonstrate the effectiveness of the proposed method.

中文翻译：

---

使用生成对抗网络的半监督跨模态图像生成

摘要 跨模态图像生成是多模态学习的一个重要方面。现有方法通常使用语义特征来减少模态差距。这些方法虽然取得了显着的进步，但仍然存在一些局限性：（1）它们通常使用单一模态信息来学习语义特征；(2) 他们要求训练数据配对。为了克服这些问题，我们提出了一种新的半监督跨模态图像生成方法，它由两个语义网络和一个图像生成网络组成。具体来说，在语义网络中，我们通过使用深度互学习策略，使用图像模态来辅助非图像模态进行语义特征学习。在图像生成网络中，我们引入了一个额外的鉴别器来减少图像重建损失。通过利用大量未配对的数据，我们的方法可以以半监督的方式进行训练。大量实验证明了所提出方法的有效性。