Recent studies have demonstrated that the auditory speech perception of a listener can be modulated by somatosensory input applied to the facial skin suggesting that perception is an embodied process. However, speech perception is a multisensory process involving both the auditory and visual modalities. It is unknown whether and to what extent somatosensory stimulation to the facial skin modulates audio-visual speech perception. If speech perception is an embodied process, then somatosensory stimulation applied to the perceiver should influence audio-visual speech processing. Using the McGurk effect (the perceptual illusion that occurs when a sound is paired with the visual representation of a different sound, resulting in the perception of a third sound) we tested the prediction using a simple behavioral paradigm and at the neural level using event-related potentials (ERPs) and their cortical sources. We recorded ERPs from 64 scalp sites in response to congruent and incongruent audio-visual speech randomly presented with and without somatosensory stimulation associated with facial skin deformation. Subjects judged whether the production was /ba/ or not under all stimulus conditions. In the congruent audio-visual condition subjects identifying the sound as /ba/, but not in the incongruent condition consistent with the McGurk effect. Concurrent somatosensory stimulation improved the ability of participants to more correctly identify the production as /ba/ relative to the non-somatosensory condition in both congruent and incongruent conditions. ERP in response to the somatosensory stimulation for the incongruent condition reliably diverged 220 msec after stimulation onset. Cortical sources were estimated around the left anterior temporal gyrus, the right middle temporal gyrus, the right posterior superior temporal lobe and the right occipital region. The results demonstrate a clear multisensory convergence of somatosensory and audio-visual processing in both behavioral and neural processing consistent with the perspective that speech perception is a self-referenced, sensorimotor process.

最近的研究表明，听众的听觉言语感知可以通过应用于面部皮肤的体感输入来调节，这表明感知是一个具体的过程。然而，语音感知是一个涉及听觉和视觉方式的多感官过程。目前尚不清楚对面部皮肤的体感刺激是否以及在多大程度上调节视听语音感知。如果语音感知是一个具体过程，那么应用于感知者的体感刺激应该会影响视听语音处理。利用麦格克效应（当声音与不同声音的视觉表示配对时发生的知觉错觉，导致第三种声音的感知），我们使用简单的行为范式并使用事件在神经层面测试了预测。相关电位（ERP）及其皮质来源。我们记录了 64 个头皮部位的 ERP，以响应随机出现的一致和不一致的视听语音，有或没有与面部皮肤变形相关的体感刺激。受试者判断在所有刺激条件下是否产生/ba/。在一致的视听条件下，受试者将声音识别为/ba/，但在与麦格克效应一致的不一致条件下，受试者则不会。并发体感刺激提高了参与者在一致和不一致条件下相对于非体感条件更正确地将产生识别为/ba/的能力。在不一致条件下对体感刺激的 ERP 响应在刺激开始后 220 毫秒可靠地发散。估计左前颞回、右颞中回、右后上颞叶和右枕区周围的皮质源。结果表明，行为和神经处理中的体感和视听处理具有明显的多感官融合，这与语音感知是一种自我参考的感觉运动过程的观点相一致。