We present two related but different cross-situational and crossmodal models of incremental word learning. Model 1 is a Bayesian approach for co-learning object-word mappings and referential intention which allows for incremental learning from only a few situations where the display of referents to the learning system is systematically varied. We demonstrate the robustness of the model with respect to sensory noise, including errors in the visual (object recognition) and auditory (recognition of words) systems. The model is then integrated with a cognitive robotic architecture in order to realize cross-situational word learning on a robot. A different approach to word learning is demonstrated with Model 2, an information-theoretic model for the object- and action-word learning from modality rich input data based on pointwise mutual information. The approach is inspired by insights from language development and learning where the caregiver/teacher typically shows objects and performs actions to the infant while naming what the teacher is doing. We demonstrate the word learning capabilities of the model, feeding it with crossmodal input data from two German multimodal corpora which comprise visual scenes of performed actions and related utterances.

中文翻译：

---

面向任务场景中跨情境和跨模态单词学习的模型

我们提出了两个相关但不同的增量词学习的跨情境和跨模态模型。模型 1 是一种贝叶斯方法，用于共同学习对象-词映射和参照意图，它允许仅从少数情况下进行增量学习，其中学习系统的参照物显示系统地变化。我们证明了模型在感官噪声方面的稳健性，包括视觉（物体识别）和听觉（单词识别）系统中的错误。然后将该模型与认知机器人架构集成，以实现机器人上的跨情境单词学习。模型 2 展示了一种不同的单词学习方法，模型 2 是一种信息理论模型，用于从基于点互信息的模态丰富的输入数据中学习对象和动作词。该方法的灵感来自语言发展和学习的见解，在这种情况下，看护者/教师通常会向婴儿展示物体并执行动作，同时说出教师正在做什么。我们展示了模型的单词学习能力，将来自两个德国多模态语料库的跨模态输入数据提供给它，这些数据包括执行动作和相关话语的视觉场景。