当前位置: X-MOL 学术Front. Neurorobotics › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Crossmodal Language Comprehension-Psycholinguistic Insights and Computational Approaches.
Frontiers in Neurorobotics ( IF 3.1 ) Pub Date : 2020-01-31 , DOI: 10.3389/fnbot.2020.00002
Özge Alaçam 1 , Xingshan Li 2 , Wolfgang Menzel 1 , Tobias Staron 1
Affiliation  

Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches.

中文翻译:

跨模态语言理解-心理语言学见解和计算方法。

情景语言理解中的跨模态交互对于有效和高效的沟通非常重要。语言和视觉刺激之间的关系提供了互惠互利:例如,视觉有助于提供信息以提高语言理解,而语言反过来又在视觉环境中推动注意力集中方面发挥作用。然而,语言和视觉是两种不同的表征方式,它们适应概念化的不同方面和粒度。将它们集成到一个单一的、连贯的系统解决方案中仍然是一个挑战,这可以从人类跨模式处理的灵感中受益。基于对情境语言理解本质的基本心理语言学见解,我们得出了一组促进语言理解稳健性的性能特征,例如跨模态参考解析、注意力引导或预测处理。用于语言理解的人工系统应该满足这些特征,以便能够以自然流畅的方式执行。我们讨论了人类语言理解的跨模式支持的实证研究结果如何应用于情境语言理解的计算解决方案,以及它们如何帮助减轻当前方法的缺点。
更新日期:2020-01-31
down
wechat
bug