We present a deep learning approach for learning the joint semantic embeddings of images and captions in a euclidean space, such that the semantic similarity is approximated by the L2L_{2} distances in the embedding space. For that, we introduce a metric learning scheme that utilizes multitask learning to learn the embedding of identical semantic concepts using a center loss. By introducing a differentiable quantization scheme into the end-to-end trainable network, we derive a semantic embedding of semantically similar concepts in euclidean space. We also propose a novel metric learning formulation using an adaptive margin hinge loss, that is refined during the training phase. The proposed scheme was applied to the MS-COCO, Flicke30K and Flickr8K datasets, and was shown to compare favorably with contemporary state-of-the-art approaches.

中文翻译：

---

学习嵌入语义相似性以进行图像-文本联合检索


我们提出了一种深度学习方法，用于学习欧几里得空间中图像和标题的联合语义嵌入，从而通过嵌入空间中的 L2L_{2} 距离来近似语义相似性。为此，我们引入了一种度量学习方案，该方案利用多任务学习来使用中心损失来学习相同语义概念的嵌入。通过将可微量化方案引入端到端可训练网络，我们在欧几里得空间中推导了语义相似概念的语义嵌入。我们还提出了一种使用自适应边缘铰链损失的新颖度量学习公式，该公式在训练阶段进行了完善。所提出的方案应用于 MS-COCO、Flicke30K 和 Flickr8K 数据集，并被证明与当代最先进的方法相比具有优势。