Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model over-fitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis.

中文翻译：

---

基于细粒度实例级草图的视频检索

现有的草图分析工作研究描绘静态对象或场景的草图。在这项工作中，我们提出了一种新的基于实例级草图的细粒度视频检索 (FG-SBVR) 的跨模态检索问题，其中草图序列用作检索特定目标视频实例的查询。与基于草图的静止图像检索和粗粒度类别级视频检索相比，这更具挑战性，因为视觉外观和运动都需要在细粒度级别同时匹配。我们贡献了第一个具有丰富注释的 FG-SBVR 数据集。然后，我们引入了一种新颖的多流多模态深度网络，以在强监督和弱监督设置下执行 FG-SBVR。该网络的关键组件是一个关系模块，旨在防止模型在缺乏训练数据的情况下过度拟合。