当前位置: X-MOL 学术BMC Med. Inform. Decis. Mak. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Monitoring stance towards vaccination in twitter messages.
BMC Medical Informatics and Decision Making ( IF 3.5 ) Pub Date : 2020-02-18 , DOI: 10.1186/s12911-020-1046-y
Florian Kunneman 1, 2 , Mattijs Lambooij 3 , Albert Wong 3 , Antal van den Bosch 1, 4 , Liesbeth Mollema 3
Affiliation  

BACKGROUND We developed a system to automatically classify stance towards vaccination in Twitter messages, with a focus on messages with a negative stance. Such a system makes it possible to monitor the ongoing stream of messages on social media, offering actionable insights into public hesitance with respect to vaccination. At the moment, such monitoring is done by means of regular sentiment analysis with a poor performance on detecting negative stance towards vaccination. For Dutch Twitter messages that mention vaccination-related key terms, we annotated their stance and feeling in relation to vaccination (provided that they referred to this topic). Subsequently, we used these coded data to train and test different machine learning set-ups. With the aim to best identify messages with a negative stance towards vaccination, we compared set-ups at an increasing dataset size and decreasing reliability, at an increasing number of categories to distinguish, and with different classification algorithms. RESULTS We found that Support Vector Machines trained on a combination of strictly and laxly labeled data with a more fine-grained labeling yielded the best result, at an F1-score of 0.36 and an Area under the ROC curve of 0.66, considerably outperforming the currently used sentiment analysis that yielded an F1-score of 0.25 and an Area under the ROC curve of 0.57. We also show that the recall of our system could be optimized to 0.60 at little loss of precision. CONCLUSION The outcomes of our study indicate that stance prediction by a computerized system only is a challenging task. Nonetheless, the model showed sufficient recall on identifying negative tweets so as to reduce the manual effort of reviewing messages. Our analysis of the data and behavior of our system suggests that an approach is needed in which the use of a larger training dataset is combined with a setting in which a human-in-the-loop provides the system with feedback on its predictions.

中文翻译:

监视推特消息中疫苗接种的立场。

背景技术我们开发了一种系统来自动对Twitter消息中的疫苗接种立场进行分类,重点是负面立场的消息。这样的系统使得有可能监视社交媒体上正在进行的消息流,提供有关疫苗接种的公众犹豫的可行见解。目前,这种监测是通过定期的情绪分析来完成的,在检测对疫苗接种的负面态度方面表现不佳。对于提及疫苗接种相关关键术语的荷兰Twitter消息,我们注释了其与疫苗接种有关的立场和感觉(前提是他们提到了该主题)。随后,我们使用这些编码数据来训练和测试不同的机器学习设置。为了最好地识别对疫苗接种持消极态度的信息,我们以增加的数据集大小和降低的可靠性,以增加的类别数量进行区分以及使用不同的分类算法来比较设置。结果我们发现,在严格和宽松标签的数据结合更细粒度标签的基础上训练的支持向量机产生了最佳结果,F1得分为0.36,ROC曲线下的面积为0.66,大大优于当前使用情感分析得出F1得分为0.25,ROC曲线下面积为0.57。我们还表明,我们的系统的召回率可以在不影响精度的情况下优化为0.60。结论我们的研究结果表明,仅通过计算机系统进行的姿势预测是一项艰巨的任务。尽管如此,该模型在识别负面推文方面显示出足够的回忆性,从而减少了查看信息的人工工作。我们对系统数据和行为的分析表明,需要一种方法,其中使用较大的训练数据集并结合使用这种方法,其中人在回路中为系统提供有关其预测的反馈。
更新日期:2020-04-22
down
wechat
bug