Human communication can be either monosemiotic or polysemiotic, depending on whether it combines ensembles of representations from one or more semiotic systems such as language, gesture and depiction. Each semiotic system has its unique storytelling potentials, which makes intersemiotic translation from one system to another challenging. We investigated the influence of the source semiotic system, realised in speech and a sequence of pictures, respectively, on the way the same story was retold using speech and co-speech gestures. The story was the content of the picture book Frog, where are you? A group of Finnish speakers saw the story in pictures, and another group heard it in matched oral narration. Each participant retold the story to an addressee and all narrations were video-recorded and analysed for both speech and gestures. Given the high degree of iconicity in depiction, we expected more iconic gestures (especially enactments) in the narratives translated from pictures than from speech. Conversely, we expected greater narrative coherence in the narratives translated from speech. The results showed that more iconic gestures were produced in the narratives translated from speech, but these were primarily not from the enactment subtype. As expected, iconic enactments were more frequent in the narratives translated from the story presented in pictures. The narratives produced by participants who had only heard the story did not have a greater variety of connective devices, but the type of devices differed slightly between the groups. Together with some additional differences between the groups that had not been anticipated, the results indicate that a story presented in different semiotic systems tend to be translated into different polysemiotic narratives. (Less)

中文翻译：

---

从单语种叙事到多语种叙事的翻译：芬兰言语和手势研究

人类交流可以是单符号的，也可以是多符号的，这取决于它是否结合了来自一个或多个符号系统（例如语言，手势和描绘）的表示形式。每个符号系统都有其独特的叙事潜力，这使得符号间翻译从一个系统转换为另一个系统具有挑战性。我们调查了分别通过语音和图片序列实现的源符号系统对使用语音和共语音手势重新讲述同一故事的方式的影响。故事是图画书青蛙的内容，你在哪里？一组芬兰讲者在图片中看到了这个故事，另一组则以相配的口头叙述听到了。每个参与者都将故事重播给收件人，并且所有旁白都被录制了视频并分析了语音和手势。鉴于描绘中的高度象似性，我们期望从图片中翻译出的叙事中的标志性手势（特别是演说）多于语音。相反，我们期望从语音翻译的叙事中具有更大的叙事连贯性。结果表明，从语音翻译的叙事中产生了更多的标志性手势，但这些手势手势主要不是来自角色亚型。不出所料，从图片故事中翻译出来的叙述中，标志性的用法更为频繁。仅听过该故事的参与者所制作的叙述并没有更多种类的连接设备，但两组之间的设备类型略有不同。加上各组之间未曾预料到的其他差异，结果表明，在不同符号系统中呈现的故事倾向于被翻译成不同的多符号叙事。（减）