Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson and Holler 2014). This article introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 29 h of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data. All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside and Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of re-enactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech). The multimodal data are derived from psychophysiological measurements and eye tracking. The psychophysiological measurements include, inter alia, electrodermal activity or GSR, which is indicative of emotional arousal (e.g. Peräkylä et al. 2015). Eye tracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the area-of-interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022), while changes in pupil size provide a window onto cognitive intensity (e.g. Barthel and Sauppe 2019). To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data open up for research in Interactional (Corpus) Linguistics, this article reports on interim results derived from work-in-progress.

中文翻译：

---

触及冰山一角：弗莱堡多模式交互语料库指南

大多数语料库默认支持仅语音视图，过滤掉任何不是“单词”的内容，并仅以拼字法转录口语，尽管事实上“仅语音视图从根本上来说是不完整的”（Kok 2017，2）言语、声音和运动方式的深度交织（Levinson 和 Holler 2014）。本文介绍了弗莱堡多模态交互语料库（FreMIC），这是一个目前正在建设中的英语即兴对话的多模态交互语料库。在撰写本文时，FreMIC 包括 (i) c。29 小时的视频记录被详细转录和注释，并且 (ii) 自动（和手动）生成多模式数据。所有对话均以正字法和杰斐逊惯例在 ELAN 中转录，以呈现语言内容和交互相关的排序细节（例如重叠、锁定）、时间方面（暂停、加速/减速）、语音方面（例如强度、音高、拉伸、截断） 、语音质量）和笑声。此外，使用 CLAWS 网络标记器对正字法转录进行了详尽的 PoS 标记（Garside 和 Smith 1997）。基于 ELAN 的转录还提供重演（也称为（自由）直接语音、构造对话等）以及无声手势（没有伴随语音的有意义的手势）的详尽注释。多模态数据来自心理生理学测量和眼球追踪。心理生理学测量尤其包括皮肤电活动或 GSR，它表示情绪唤醒（例如 Peräkylä 等人，2015 年）。眼球追踪产生两种数据：注视方向和瞳孔大小。在 FreMIC 中，使用感兴趣区域技术自动记录注视。凝视方向是交互的关键，例如，在轮流（例如 Auer 2021）和重演（例如 Pfeiffer 和 Weiss 2022）中，而瞳孔大小的变化提供了了解认知强度的窗口（例如 Barthel 和 Sauppe 2019）。为了展示 FreMIC 的转录、注释和多模态数据（组合）为交互（语料库）语言学研究带来的机会，本文报告了正在进行的工作的中期结果。