Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

在这里，我们研究了视觉空间工作记忆 (WM) 在理解具有共同语音标志性手势的多模态话语过程中的作用。EEG 被记录为健康成年人在网格上编码一个（低负载）或四个（高负载）点位置的序列并排练它们，直到在试验后期收集到自由回忆响应。在 WM 任务的排练期间，参与者观察了演讲者描述对象的视频，其中一半的试验包括语义相关的共同语音手势（一致），另一半包括语义无关的手势（不一致）。话语处理由视频期间 alpha 和 beta 波段中的振荡 EEG 活动索引。在所有参与者中，言语和手势不一致的影响在低负荷试验中比在高负荷试验中更为明显。视觉空间 WM 容量的个体差异也会调节效果。这些数据表明视觉空间 WM 资源是在理解多模态话语时被招募的。