Abstract

Objective

Automated analysis of vaccine postmarketing surveillance narrative reports is important to understand the progression of rare but severe vaccine adverse events (AEs). This study implemented and evaluated state-of-the-art deep learning algorithms for named entity recognition to extract nervous system disorder-related events from vaccine safety reports.

Materials and Methods

We collected Guillain-Barré syndrome (GBS) related influenza vaccine safety reports from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016. VAERS reports were selected and manually annotated with major entities related to nervous system disorders, including, investigation, nervous_AE, other_AE, procedure, social_circumstance, and temporal_expression. A variety of conventional machine learning and deep learning algorithms were then evaluated for the extraction of the above entities. We further pretrained domain-specific BERT (Bidirectional Encoder Representations from Transformers) using VAERS reports (VAERS BERT) and compared its performance with existing models.

Results and Conclusions

Ninety-one VAERS reports were annotated, resulting in 2512 entities. The corpus was made publicly available to promote community efforts on vaccine AEs identification. Deep learning-based methods (eg, bi-long short-term memory and BERT models) outperformed conventional machine learning-based methods (ie, conditional random fields with extensive features). The BioBERT large model achieved the highest exact match F-1 scores on nervous_AE, procedure, social_circumstance, and temporal_expression; while VAERS BERT large models achieved the highest exact match F-1 scores on investigation and other_AE. An ensemble of these 2 models achieved the highest exact match microaveraged F-1 score at 0.6802 and the second highest lenient match microaveraged F-1 score at 0.8078 among peer models.

中文翻译：

---

使用深度学习从疫苗不良事件报告系统 (VAERS) 中的安全报告中提取上市后不良事件

摘要

客观的

疫苗上市后监测叙述报告的自动分析对于了解罕见但严重的疫苗不良事件 (AE) 的进展非常重要。本研究实施并评估了用于命名实体识别的最先进的深度学习算法，以从疫苗安全报告中提取与神经系统疾病相关的事件。

材料和方法

我们从 1990 年至 2016 年从疫苗不良事件报告系统 (VAERS) 收集了与格林-巴利综合征 (GBS) 相关的流感疫苗安全性报告。选择了 VAERS 报告并手动注释了与神经系统疾病相关的主要实体，包括调查、紧张_AE、other_AE、过程、social_circumstance和temporal_expression。然后评估了各种传统的机器学习和深度学习算法以提取上述实体。我们使用 VAERS 报告 (VAERS BERT) 进一步预训练了特定领域的 BERT（来自 Transformers 的双向编码器表示），并将其性能与现有模型进行了比较。

结果和结论

对 91 份 VAERS 报告进行了注释，产生了 2512 个实体。该语料库已公开发布，以促进社区在疫苗 AE 识别方面的努力。基于深度学习的方法（例如，双长短期记忆和 BERT 模型）优于传统的基于机器学习的方法（即，具有广泛特征的条件随机场）。BioBERT 大型模型在neuro_AE、procedure、social_circumstance和temporal_expression上获得了最高的精确匹配 F-1 分数；而 VAERS BERT 大型模型在调查和其他方面取得了最高的精确匹配 F-1 分数_AE. 这 2 个模型的集合在同类模型中获得了最高的精确匹配微平均 F-1 分数，为 0.6802，第二高的宽松匹配微平均 F-1 分数为 0.8078。