The purpose of cross-modal retrieval is to find the relationship between different modal samples and to retrieve other modal samples with similar semantics by using a certain modal sample. As the data of different modalities presents heterogeneous low-level feature and semantic-related high-level features, the main problem of cross-modal retrieval is how to measure the similarity between different modalities. In this article, we present a novel cross-modal retrieval method, named Hybrid Cross-Modal Similarity Learning model (HCMSL for short). It aims to capture sufficient semantic information from both labeled and unlabeled cross-modal pairs and intra-modal pairs with same classification label. Specifically, a coupled deep fully connected networks are used to map cross-modal feature representations into a common subspace. Weight-sharing strategy is utilized between two branches of networks to diminish cross-modal heterogeneity. Furthermore, two Siamese CNN models are employed to learn intra-modal similarity from samples of same modality. Comprehensive experiments on real datasets clearly demonstrate that our proposed technique achieves substantial improvements over the state-of-the-art cross-modal retrieval techniques.

中文翻译：

---

HCMSL：用于跨模态检索的混合跨模态相似性学习

跨模态检索的目的是寻找不同模态样本之间的关系，并利用某个模态样本检索其他语义相似的模态样本。由于不同模态的数据呈现异构的低级特征和语义相关的高级特征，跨模态检索的主要问题是如何衡量不同模态之间的相似性。在本文中，我们提出了一种新的跨模态检索方法，称为混合跨模态相似性学习模型（简称 HCMSL）。它旨在从标记和未标记的跨模态对以及具有相同分类标签的模态对中捕获足够的语义信息。具体来说，耦合深度全连接网络用于将跨模态特征表示映射到公共子空间。在网络的两个分支之间使用权重共享策略来减少跨模式异质性。此外，两个连体 CNN 模型用于从相同模态的样本中学习模态内相似性。对真实数据集的综合实验清楚地表明，我们提出的技术比最先进的跨模态检索技术实现了实质性改进。