We present a novel concept audio–visual object removal in 360-degree videos, in which a target object in a 360-degree video is removed in both the visual and auditory domains synchronously. Previous methods have solely focused on the visual aspect of object removal using video inpainting techniques, resulting in videos with unreasonable remaining sounds corresponding to the removed objects. We propose a solution which incorporates direction acquired during the video inpainting process into the audio removal process. More specifically, our method identifies the sound corresponding to the visually tracked target object and then synthesizes a three-dimensional sound field by subtracting the identified sound from the input 360-degree video. We conducted a user study showing that our multi-modal object removal supporting both visual and auditory domains could significantly improve the virtual reality experience, and our method could generate sufficiently synchronous, natural and satisfactory 360-degree videos.

中文翻译：

---

360 度视频中的视听对象移除

我们提出了 360 度视频中视听对象去除的新概念，其中 360 度视频中的目标对象在视觉和听觉域中同步去除。以前的方法只关注使用视频修复技术去除对象的视觉方面，导致视频具有与被删除对象相对应的不合理的剩余声音。我们提出了一种解决方案，将视频修复过程中获得的方向融入音频去除过程中。更具体地说，我们的方法识别与视觉跟踪的目标对象对应的声音，然后通过从输入的 360 度视频中减去识别出的声音来合成 3 维声场。