In 2019, outbreaks of vaccine-preventable diseases reached the highest number in the US since 1992. Medical misinformation, such as antivaccine content propagating through social media, is associated with increases in vaccine delay and refusal. Our overall goal is to develop an automatic detector for antivaccine messages to counteract the negative impact that antivaccine messages have on the public health. Very few extant detection systems have considered multimodality of social media posts (images, texts, and hashtags), and instead focus on textual components, despite the rapid growth of photo-sharing applications (e.g., Instagram). As a result, existing systems are not sufficient for detecting antivaccine messages with heavy visual components (e.g., images) posted on these newer platforms. To solve this problem, we propose a deep learning network that leverages both visual and textual information. A new semantic- and task-level attention mechanism was created to help our model to focus on the essential contents of a post that signal antivaccine messages. The proposed model, which consists of three branches, can generate comprehensive fused features for predictions. Moreover, an ensemble method is proposed to further improve the final prediction accuracy. To evaluate the proposed model's performance, a real-world social media dataset that consists of more than 30,000 samples was collected from Instagram between January 2016 and October 2019. Our 30 experiment results demonstrate that the final network achieves above 97% testing accuracy and outperforms other relevant models, demonstrating that it can detect a large amount of antivaccine messages posted daily. The implementation code is available at https://github.com/wzhings/antivaccine_detection.

中文翻译：

---

使用多模式深度学习在社交媒体上检测医学错误信息

2019年，疫苗可预防疾病的暴发达到了1992年以来美国的最高数量。医学错误信息，例如通过社交媒体传播的抗疫苗含量，与疫苗延误和拒绝增加有关。我们的总体目标是开发一种自动检测抗疫苗信息的检测器，以抵消抗疫苗信息对公众健康的负面影响。尽管照片共享应用程序（例如，Instagram）迅速增长，但很少有现成的检测系统考虑过社交媒体帖子（图像，文本和主题标签）的多模式性，而是专注于文本组件。结果，现有系统不足以检测带有张贴在这些较新平台上的大量视觉组件（例如图像）的抗疫苗消息。为了解决这个问题，我们提出了一个利用视觉和文字信息的深度学习网络。创建了一个新的语义和任务级别的注意机制，以帮助我们的模型将重点放在发给抗疫苗信息的帖子的基本内容上。所提出的模型由三个分支组成，可以生成用于预测的综合融合特征。此外，提出了一种集成方法以进一步提高最终预测精度。为了评估建议模型的性能，在2016年1月至2019年10月之间，从Instagram收集了一个由30,000多个样本组成的真实世界社交媒体数据集。我们的30个实验结果表明，最终网络的测试准确率达到97％以上，并且优于其他网络相关模型 证明它可以检测到每天发布的大量抗疫苗消息。实施代码可从https://github.com/wzhings/antivaccine_detection获得。