Background: The wide adoption of social media in daily life renders it a rich and effective resource for conducting near real-time assessments of consumers’ perceptions of health services. However, its use in these assessments can be challenging because of the vast amount of data and the diversity of content in social media chatter. Objective: This study aims to develop and evaluate an automatic system involving natural language processing and machine learning to automatically characterize user-posted Twitter data about health services using Medicaid, the single largest source of health coverage in the United States, as an example. Methods: We collected data from Twitter in two ways: via the public streaming application programming interface using Medicaid-related keywords (Corpus 1) and by using the website’s search option for tweets mentioning agency-specific handles (Corpus 2). We manually labeled a sample of tweets in 5 predetermined categories or other and artificially increased the number of training posts from specific low-frequency categories. Using the manually labeled data, we trained and evaluated several supervised learning algorithms, including support vector machine, random forest (RF), naïve Bayes, shallow neural network (NN), k-nearest neighbor, bidirectional long short-term memory, and bidirectional encoder representations from transformers (BERT). We then applied the best-performing classifier to the collected tweets for postclassification analyses to assess the utility of our methods. Results: We manually annotated 11,379 tweets (Corpus 1: 9179; Corpus 2: 2200) and used 7930 (69.7%) for training, 1449 (12.7%) for validation, and 2000 (17.6%) for testing. A classifier based on BERT obtained the highest accuracies (81.7%, Corpus 1; 80.7%, Corpus 2) and F1 scores on consumer feedback (0.58, Corpus 1; 0.90, Corpus 2), outperforming the second best classifiers in terms of accuracy (74.6%, RF on Corpus 1; 69.4%, RF on Corpus 2) and F1 score on consumer feedback (0.44, NN on Corpus 1; 0.82, RF on Corpus 2). Postclassification analyses revealed differing intercorpora distributions of tweet categories, with political (400778/628411, 63.78%) and consumer feedback (15073/27337, 55.14%) tweets being the most frequent for Corpus 1 and Corpus 2, respectively. Conclusions: The broad and variable content of Medicaid-related tweets necessitates automatic categorization to identify topic-relevant posts. Our proposed system presents a feasible solution for automatic categorization and can be deployed and generalized for health service programs other than Medicaid. Annotated data and methods are available for future studies. Trial Registration:

This is the abstract only. Read the full article on the JMIR site. JMIR is the leading open access journal for eHealth and healthcare in the Internet age.

中文翻译：

---

开发一个自动系统，用于对 Twitter 上关于健康服务的聊天进行分类：医疗补助的案例研究

背景：社交媒体在日常生活中的广泛采用使其成为对消费者对健康服务的看法进行近实时评估的丰富而有效的资源。然而，由于社交媒体聊天中的大量数据和内容的多样性，它在这些评估中的使用可能具有挑战性。目标：本研究旨在开发和评估一个涉及自然语言处理和机器学习的自动系统，以使用美国最大的单一健康保险来源 Medicaid 为例，自动表征用户发布的有关健康服务的 Twitter 数据。方法：我们通过两种方式从 Twitter 收集数据：通过公共流媒体应用程序编程接口使用与医疗补助相关的关键字（语料库 1）和使用网站的搜索选项来搜索提及机构特定句柄的推文（语料库 2）。我们手动标记了 5 个预定类别或其他类别的推文样本，并人为地增加了来自特定低频类别的训练帖子的数量。使用手动标记的数据，我们训练和评估了几种监督学习算法，包括支持向量机、随机森林 (RF)、朴素贝叶斯、浅层神经网络 (NN)、k-最近邻、双向长短期记忆和双向来自转换器 (BERT) 的编码器表示。然后，我们将性能最佳的分类器应用于收集的推文进行分类后分析，以评估我们方法的效用。结果：我们手动注释了 11,379 条推文（语料库 1：9179；语料库 2：2200）并使用 7930（69.7%）条进行训练，1449 条（12.7%）用于验证，2000 条（17.6%）用于测试。基于 BERT 的分类器获得了最高准确率（81.7%，语料库 1；80.7%，语料库 2）和消费者反馈的 F1 分数（0.58，语料库 1；0.90，语料库 2），在准确度方面优于第二好的分类器（ 74.6%，语料库 1 的 RF；69.4%，语料库 2 的 RF）和消费者反馈的 F1 分数（0.44，语料库 1 的 NN；0.82，语料库 2 的 RF）。分类后分析揭示了推文类别的不同语料库间分布，政治 (400778/628411, 63.78%) 和消费者反馈 (15073/27337, 55.14%) 推文分别是语料库 1 和语料库 2 中最常见的推文。结论：与医疗补助相关的推文内容广泛且多变，因此需要自动分类以识别与主题相关的帖子。我们提出的系统为自动分类提供了一种可行的解决方案，可以部署和推广到医疗补助以外的健康服务项目。注释数据和方法可用于未来的研究。试用注册：

这只是摘要。阅读 JMIR 网站上的完整文章。JMIR 是互联网时代电子健康和医疗保健领域领先的开放获取期刊。