This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced. For that, we propose a multimodal approach to simultaneous machine translation using reinforcement learning, with strategies to integrate visual and textual information in both the agent and the environment. We provide an exploration on how different types of visual information and integration strategies affect the quality and latency of simultaneous translation models, and demonstrate that visual cues lead to higher quality while keeping the latency low.

中文翻译：

---

利用多模式强化学习进行同时机器翻译

本文通过探讨两个主要概念解决了同时机器翻译（SiMT）的问题：（a）自适应策略，以学习高翻译质量和低延迟之间的良好权衡；（b）通过提供可能产生文本输入之前可用的其他（视觉）上下文信息来支持此过程的视觉信息。为此，我们提出了一种采用强化学习的同时机器翻译的多模式方法，该方法具有在代理和环境中集成可视和文本信息的策略。我们提供了有关不同类型的视觉信息和集成策略如何影响同步翻译模型的质量和延迟的探索，并证明了视觉提示可以在降低延迟的同时提高质量。