This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.

中文翻译：

---

隐喻加工是超模态语义加工：双侧颞区在多模态交流中的作用

本文介绍了一项关于健康成人对多模态交流中隐喻理解的 fMRI 研究。我们研究了仅以隐喻手势（“单模态隐喻”）或伴随手势（“多模态隐喻”）表达的隐喻。单模态隐喻手势传达了伴随语音中未表达的隐喻信息（例如说非隐喻的话语，“她感觉不好”，同时手掌朝上放下手；在这里，手势单独表示隐喻），而多模态中的隐喻手势隐喻表示言语中多余的隐喻（例如说比喻性的话语，“她的精神下降了”，同时手掌朝上放下手）。换句话说，在单模态隐喻中，手势添加了未说出来的信息，而多模态隐喻中的手势对于口语内容可能是多余的。理解和整合每种模态中的信息，这里是口语和视觉，在多模态交流中很重要，但大多数先前的研究只考虑了多模态隐喻，其中手势与所说的内容是多余的。我们的参与者观看了演员一边打手势一边讲话的视听剪辑。我们发现，无论概念隐喻的表达方式如何，抽象隐喻理解都会招募外侧上/中颞叶皮层。这些结果表明，无论模态如何，抽象隐喻都涉及一般语义加工中涉及的资源，并且与这些领域在超模态语义加工中的作用以及具身认知理论是一致的。