Abstract

Practitioners and scholars increasingly employ social media listening platforms (SMLPs) driven by artificial intelligence (AI) to extract actionable insights from large amounts of social media data informing research questions and brand strategy. Due to their proprietary nature, AI tools within SMLPs are “black boxes” that force users to accept results on blind faith, a source of concern in industry and academia. This study seeks to provide greater understanding of the strengths and weaknesses of SMLPs by assessing the AI-based results of leading SMLP Crimson Hexagon (now Brandwatch Consumer Research) against those of a standard human content analysis and an analysis conducted using Linguistic Inquiry and Word Count (LIWC). Analyzing a random 10,000-post sample of the conversation around the Nike “Dream Crazy” ad featuring Colin Kaepernick, findings reveal Crimson Hexagon’s AI tools to be woefully unreliable in terms of brand identification as well as detection of post and brand sentiment polarity, specific emotions, and brand outcomes, demonstrating the hazards of blindly relying upon conclusions drawn from black-box social media listening platforms. Findings highlight the need for researchers to examine algorithm documentation and training data sets, as well as assess AI-generated data prior to use in research models and decisions.

摘要

从业者和学者越来越多地使用由人工智能（AI）驱动的社交媒体监听平台（SMLP），从大量的社交媒体数据中提取可行的见解，以告知研究问题和品牌策略。由于其专有性质，SMLP中的AI工具是“黑匣子”，迫使用户接受盲目的结果，这是行业和学术界的关注之源。这项研究旨在通过评估领先的SMLP Crimson Hexagon（现为Brandwatch Consumer Research）基于AI的结果与标准的人类内容分析以及使用语言查询和字数统计进行的分析，来提供对SMLP优缺点的更多理解（LIWC）。分析了以Colin Kaepernick为特色的Nike“ Dream Crazy”广告周围的随机10,000个帖子示例，调查结果表明，Crimson Hexagon的AI工具在品牌识别以及职位和品牌情感极性，特定情感和品牌结果的检测方面非常不可靠，表明了盲目依赖黑盒社交媒体收听平台得出的结论的危害。研究结果强调研究人员需要检查算法文档和训练数据集，以及在将AI生成的数据用于研究模型和决策之前进行评估。