This study explores the potential of including AI-generated language in pragmatic analysis– a field that has primarily been conducted on human language use. With the rapid growth of large language models, AI-generated texts and AI-human interactions constitute a growing field where pragmatics research is expanding to. Language data that humans used to hold a full authorship may also involve modifications made by AI. The foremost concern is thus the pragmatic qualities of AI-generated language, such as whether and to which extent AI data mirror the pragmatic patterns we have found in human speech behaviours. In this study, we compare 148 ChatGPT-generated conversations with 82 human-written ones and 354 human evaluations of these conversations. The data are analysed using various methods, including traditional speech strategy coding, four computational methods developed in NLP, and four statistical tests. The findings reveal that ChatGPT performs equally well as human participants in four out of the five tested pragmalinguistic features and five out of six sociopragmatic features. Additionally, the conversations generated by ChatGPT exhibit higher syntactic diversity and a greater sense of formality compared to those written by humans. As a result, our participants are unable to distinguish ChatGPT-generated conversations from human-written ones.

中文翻译：

---

人工智能生成对话在语用分析中应用的可行性研究

这项研究探讨了将人工智能生成的语言纳入语用分析的潜力——这一领域主要针对人类语言使用进行。随着大型语言模型的快速增长，人工智能生成的文本和人工智能与人类的交互构成了语用学研究不断扩展的领域。人类曾经拥有完整作者权的语言数据也可能涉及人工智能所做的修改。因此，最值得关注的是人工智能生成语言的实用品质，例如人工智能数据是否以及在多大程度上反映了我们在人类言语行为中发现的实用模式。在这项研究中，我们将 148 个 ChatGPT 生成的对话与 82 个人类编写的对话以及对这些对话的 354 个人类评估进行了比较。使用各种方法对数据进行分析，包括传统的语音策略编码、NLP 中开发的四种计算方法以及四种统计测试。研究结果表明，ChatGPT 在五个测试的语用语言特征中的四个和六个社会语用特征中的五个方面与人类参与者表现相同。此外，与人类编写的对话相比，ChatGPT 生成的对话表现出更高的句法多样性和更强的正式感。因此，我们的参与者无法区分 ChatGPT 生成的对话和人类编写的对话。