DeepCADRME: A deep neural model for complex adverse drug reaction mentions extraction

Highlights

• We propose a deep neural model for complex ADR mentions extraction called DeepCADRME.

• DeepCADRME casts the problem as an N-level tagging sequence.

• It integrates an N-level model based on deep bidirectional transformer.

• The obtained results are significant compared with state-of-the-art systems.

Abstract

Extracting mentions of Adverse Drug Reaction (ADR) from biomedical texts, aiming to support pharmacovigilance and drug safety surveillance, remains a challenging task as many ADR mentions are nested, discontinuous and overlapping. To solve these issues, in this paper, we propose a deep neural model for Complex Adverse Drug Reaction Mentions Extraction, called DeepCADRME. It first transforms the ADR mentions extraction problem as an N-level tagging sequence. Then, it feeds the sequences to an N-level model based on contextual embeddings where the output of the pre-trained model of the current level is used to build a new deep contextualized representation for the next level. This allows the DeepCADRME system to transfer knowledge between levels. Experimental results performed on the TAC 2017 ADR dataset, show the effectiveness of DeepCADRME which leads to a new state-of-the-art performance by reaching a F1 of 85.35% and 85.41% with and without mention types, respectively. The evaluation results also highlight the benefits of exploring language model to effectively extract different types of ADR mentions.

Introduction

Adverse Drug Reaction (ADR), a serious public health issue, has become the leading cause of morbidity, mortality and substantial healthcare costs [6], [7], [30]. Therefore, early detection of ADR associated with drugs is a crucial challenge for pharmacovigilance. Practically, serious ADRs occur during pre-approval clinical trials. However, it is quite difficult to comprehensively characterize all ADRs, especially rare ones, due to the lack of enrolled patient diversity with relatively short duration [11]. As a result, spontaneous reporting systems present an alternative source during post-marketing surveillance to detect unobserved ADR from clinical trials. However, studies have shown that more than 90% of ADR are under-reported in such systems [20]. Natural language processing (NLP) techniques towards recognizing ADRs from clinical reports, drug labels and electronic health records provide a significant opportunity for pharmacovigilance and drug safety monitoring. In this context, the 2017 Text Analysis Conference (TAC) organized a shared track called “adverse drug reaction extraction from drug labels” [19]. The latter aimed at testing various natural language processing (NLP) and machine learning approaches for extracting ADR information from drug labels. The ADR track includes four tasks. The purpose of the first task is to extract Adverse Reaction mentions and five related mentions. The second task identifies the relations between Adverse Reaction and related mentions. The third task identifies the positive Adverse Reaction mention names within a drug label. Finally, the fourth task provides the Medical Dictionary for Regulatory Activities (MedDRA) coding for each positive Adverse Reaction identified in the third task. In this work, we are interested to handle the first task for identifying the ADR mentions from drug labels.

The task of extracting ADR mentions can be considered as a biomedical named entity recognition (BNER) problem [14]. The BNER has shown a growing interest in many text mining applications such as information retrieval [8] and question answering [22], [23], [24], [25], [26]. The detection of ADR mentions is one of the most important task of ADR systems as the overall performance of such systems is heavily depending on the effectiveness of the integrated ADR mentions extraction system: if an ADR mentions extraction system fails to identify ADR mentions, further processing steps to extract potential relationships between them will inevitably fail too. To this end, several systems have been presented to identify ADR mentions from drug labels. For instance, Gu et al. [9] proposed a BUPT-PRIS system based on Bi-Directional Long-Short Term memory (Bi-LSTM) with Conditional Random Fields (CRFs). Tao et al. [27] developed a system called PRNA-SUNY which is based on three CRFs with various features such as normalized tokens, dictionaries built from VigiBase, Unified Medical Language System (UMLS) and word embeddings. Martínez et al. [15] proposed a hybrid approach named MC-UC3M which combines a machine learning based, dictionary-based and rule-based methods. They first used a Support Vector Machine (SVM) classifier and a dictionary obtained from the Side Effect Resource (SIDER) and MedDRA to recognize ADR mentions. Then, they designed some rules to identify modifier concepts. Cocos and Masino [2] developed a CHOP system based on two modules in a pipelined process: a rule-based table extraction module and a Bi-LSTM with fastText embeddings trained on MEDLINE module. The CONDL team [28] proposed a hybrid approach based on Bi-LSTM-CRF and dictionary-based methods. The first one combines the character embedding with CNN, the word embedding with Word2Vec and one-hot encoded case embedding as features, while the second uses the SciMiner by adding the MedDRA preferred and lowest level terms. Pawar et al. [17] designed a system based on the all word pairs neural network model (AWP-NN). Belousov et al. [1] proposed a system based on Bi-LSTM model with word embedding trained on large text corpora and CRF model with UMLS semantic types, lexicon match and word lemmas as features.

Most of the aforementioned systems handled the first task as a sequence labeling problem by only extracting ADR mentions that satisfy the two conditions: (1) a mention should be contained single or sequence of words on the sentence; (2) a mention should not be embedded with other mentions. However, such conditions are perhaps nugatory in some situations as many ADR mentions are overlapping, discontinuous and nested. We call these kinds of mentions as complex ADR mentions which are described below:

• Discontinuous ADR mentions: contain at least one gap. Consider the example (a) in Fig. 1, “infusion reactions” is an ADR discontinuous mentions since it is interrupted by “of glycopeptide antibacterial agents can cause”.

• Nested ADR mentions: are embedded in other ones. For instance, given the example (b) in Fig. 1, “mild to moderate” is a Severity mention, and it contains two other mentions “mild” and “moderate”, which are also Severity.

• Overlapping ADR mentions: some words are shared by multiple mentions. Consider the example (c) in Fig. 1, the two ADR mentions “interstitial infiltration of the chest” and “Interstitial inflammation of the chest” share a common words: “interstitial” and “of the chest”. The two mentions are also discontinuous.

The complex ADR mentions cannot be recognized directly by conventional sequence labeling models which can hinder performance of such systems as they chosen to skirt the issue entirely by ignoring these mentions. On the other hand, there are only few systems that have addressed the issue of complex ADR mentions. Xu et al. [31] developed a cascaded system called UTH-CCB which applies two Bi-LSTM-CRF models with word embedding. The first model recognizes all Adverse Reaction mentions while the second one extracts the relations between the Adverse Reaction mentions and the modifier concept. The system used additional features to handle complex mentions by incorporating regular expression rules and dictionary matching results obtained from MedDRA terms. IBM Research team [3] proposed an attention-based Bi-LSTM sub-model which handles complex mentions by using a BIOHD [13] segment representation. They also used a Bi-LSTM-CRF sub-model to identify contiguous mentions by integrating word, POS and character embeddings. Nevertheless, these systems are not able to address some ambiguous situations. For instance, given the example (d) in Fig. 1, the BIOHD segment cannot extract the continuous mention “grade 3” from “grade 3 or 4” mention since the later contains both overlapping and discontinuous mentions.

To overcome the aforementioned drawbacks, in this paper, we propose a deep neural model for Complex Adverse Drug Reaction Mentions Extraction, called DeepCADRME. The motivation behind the proposed system is two-fold: (1) DeepCADRME converts the complex ADR extraction problem as an N-level tagging sequence. In contrast to the existing methods, DeepCADRME is able to cover various kinds of complexity by modeling the nested and overlapping mentions in different levels. Each level extracts ADR mentions at corresponding nested level, which is intuitive to the complex ADR mention nature. (2) DeepCADRME constructs an N-level model based on transfer learning. The N-level model builds a deep contextualized embedding at the current level for exchanging knowledge to the succeeding levels. Theoretically, the pre-trained word embeddings have contributed greatly for many areas of NLP [10]. Recently, however, more advanced embedding methods based on the transformers architecture [29] have further revolutionized the NLP field pushing state-of-art results, yet there are no attempts for how to incorporate these representations for extracting complex ADR mentions. Consequently, the purpose of this paper is to explore contextual embedding trained on various corpora to the N-level model for extracting complex ADRs, as well as, comparing these with traditional word embedding methods. Experimental results on the datasets provided by the TAC 2017 ADR challenge prove the benefit of exploring language models pre-trained on biomedical corpora to the DeepCADRME system as it effectively extracts both complex and simple ADR mentions and improves performance over state-of-the-art systems.