Recently, the witness of the rapidly growing popularity of short videos on different Internet platforms has intensified the need for a background music (BGM) retrieval system. However, existing video-music retrieval methods only based on the visual modality cannot show promising performance regarding videos with fine-grained virtual contents. In this paper, we also investigate the widely added voice-overs in short videos and propose a novel framework to retrieve BGM for fine-grained short videos. In our framework, we use the self-attention (SA) and the cross-modal attention (CMA) modules to explore the intra- and the inter-relationships of different modalities respectively. For balancing the modalities, we dynamically assign different weights to the modal features via a fusion gate. For paring the query and the BGM embeddings, we introduce a triplet pseudo-label loss to constrain the semantics of the modal embeddings. As there are no existing virtual-content video-BGM retrieval datasets, we build and release two virtual-content video datasets HoK400 and CFM400. Experimental results show that our method achieves superior performance and outperforms other state-of-the-art methods with large margins.

中文翻译：

---

利用跨模态编码的画外音检索细粒度视频的深度音乐

近来，在不同的Internet平台上短视频迅速流行的见证人越来越强烈地需要背景音乐（BGM）检索系统。但是，现有的仅基于视觉模态的视频音乐检索方法对于具有细粒度虚拟内容的视频无法表现出令人满意的性能。在本文中，我们还研究了短视频中广泛添加的旁白，并提出了一种新颖的框架来检索细粒度短视频的背景音乐。在我们的框架中，我们使用自我注意（SA）和交叉模式注意（CMA）模块分别探索不同模式的内部关系和相互关系。为了平衡模态，我们通过融合门为模态特征动态分配了不同的权重。为了解析查询和BGM嵌入，我们引入三元组伪标签丢失来约束模式嵌入的语义。由于没有现有的虚拟内容视频BGM检索数据集，因此我们构建并发布了两个虚拟内容视频数据集HoK400和CFM400。实验结果表明，我们的方法具有出色的性能，并且以较大的幅度优于其他最新方法。