We study a task of aligning time-sync video comments (danmaku) to narrative video storylines, which is helpful for finding semantic segmentation of videos and conducting fine-grained user feedback analyses. Due to the mismatch of text styles and the shift of topics, it is hard to apply previous semantic matching models directly for the alignment. To tackle the problem, we propose to utilize variational auto-encoders to map both user comments and storylines into latent spaces. By posing a matching loss on their latent codes, we reduce their mismatches in the latent space and make the alignment easier to learn. To handle constraints in the alignment, we also apply dynamic programming for finding global optimal outputs. According to experiments on a TV series dataset (consisting of about 10 K pairs of storylines and danmaku sent by users), the proposed model can achieve competitive performances.

我们研究了将时间同步视频评论（danmaku）与叙事视频故事情节对齐的任务，这有​​助于查找视频的语义细分并进行细粒度的用户反馈分析。由于文本样式的不匹配和主题的变化，很难直接将以前的语义匹配模型应用于对齐。为了解决该问题，我们建议利用变体自动编码器将用户评论和故事情节映射到潜在空间中。通过在它们的潜在代码上造成匹配损失，我们减少了它们在潜在空间中的不匹配，并使对齐更容易学习。为了处理对齐中的约束，我们还应用动态编程来查找全局最优输出。根据电视连续剧数据集上的实验（由用户发送的大约1万对故事情节和淡弹组成），