Joint extraction of entities and relations using graph convolution over pruned dependency trees

Abstract

We present a novel end-to-end deep neural network model based on graph convolutional networks for simultaneous joint extraction of entities and relations among them. Our model captures context and syntactic information from sentence by stacking a graph convolutional layer over bidirectional sequential LSTM layers. We sequentially concatenate the subject, object and sentence representations for obtaining the directionality of relations. Besides, in order to address long entity-distances problem, we further apply a path-centric pruning procedure to input trees in order to preserve useful information while maximally removing irrelevant words. Experiments are conducted on NYT dataset, and the proposed model achieves the state-of-the-art results on entity and relation extraction task. Our source code is available on Github: https://github.com/michael-hon/LSTM-GCN-ER.

Introduction

The task of end-to-end extraction of entities and their relations is to detect entity mentions and identify semantic relations among entities from plain text at the same time. It is the basis of many downstream natural language processing tasks, including question answering [12], knowledge graph construction [26], and summarization [2].

Traditional approaches treat this entity and relationship extraction task as a pipeline of two sub-tasks, i.e., detecting the entities [16] first and then extracting the semantic relations [43] among entities. This pipeline framework is easy and flexible since we can be free to replace one component without considering another one. However, the pipeline approach suffers from error propagation since entity recognition errors can be propagated to the relation extraction step, resulting in poor performance. In addition to this, the pipeline method also ignores the correlation between these two tasks. For instance, in the sentence Qinghai is one of the most rugged areas in China, entity Qinghai and China information is very important for correct extraction of the relation contains, and vice versa.

Recently, more and more studies focus on extracting entities and relations simultaneously in a single model, which are usually named as joint extraction method. Joint model can utilize the information of entities and relations together to achieve better performance compared to the pipeline method. [25], [18], [30] employ feature engineering to construct a joint extraction system. They require a lot of time to construct features manually and rely heavily on other NLP tools, which causes the performance of the model to depend on the features of manual design. Therefore, in order to reduce manual work time, researchers try to apply neural networks to jointly extract entities and relations in a single model [13], [45], [1], [34]. However, most end-to-end models do not consider the issue of directionality of relations, i.e., properly assigning the subject and object within a relationship. In the above Qinghai example, (China, Qinghai) hold the contains relation, while (Qinghai, China) have the administrative_divisions relationship. These two triplets (China, contains, Qinghai) and (Qinghai, administrative_divisions, China) are said to exhibit SingleEntityOverlap because they have an overlapped entity. Thus, we can treat the relation directionality as SingleEntityOverlap issue.

Besides, in order to address long entity-distance problem in long sentences, previous works [24], [19], [8] utilize the dependency tree from input sentence to extract relation with Tree-LSTM[35] or recursive neural network[32]. However, these models have some shortcomings. Since models are operating directly on the dependency tree, it is hard to parallelize and thus the computation speed becomes very slow. Moreover, some research [4], [40], [39] just utilize the shortest dependency path (SDP) between entities for removing much irrelevant information. However, only considering the shortest dependency path between entities in the dependency tree may ignore some important information, which may have some limits. For example, in Fig. 1, if the model only uses the SDP as input, token not will be not taken into account, which will cause classification error.

Based on the above analysis, we present a novel end-to-end deep neural network model to jointly extract entities and relations. Our network model mainly consists of bidirectional sequential LSTM layers and a graph convolutional layer [22], [43]. Our model first decodes entities by conditional random field (CRF) and then encodes the dependency tree over input sentences with graph convolutional layers for detecting relations between two entities. Since graph convolutional operation can use matrix multiplications, which can be easily implemented by batch training and parallel computing, and thus our model is more efficient compared with [35], [24], [19]. Besides, we sequentially concatenate the subject and object representations with sentence representation and thus the final hidden representation is different for entity pairs (e1, e2) and (e2, e1) even for the same sentence. This indicates that our model is sensitive to the directionality of relations. In order to solve the problem that the $\mathit{shortest}$ $\mathit{dependency}$ $\mathit{path}$ may lose important information, we employ a path-centric pruning technique to prune the dependency tree to maximally keep important information while removing irrelevant tokens [43].

We evaluate our model on the New York Times (NYT) dataset which is produced by distant supervised method [30]. Our model outperforms previous feature-based and neural network-based methods, which indicates the effectiveness of our proposed model. Besides, the experimental results show that our model can effectively address long distance between entities and the issue of relation directionality.

In summary, the main contributions of our work are: (i) we propose an end-to-end neural network model where graph convolutional network is introduced to detect relations. (ii) we consider the directionality of relations problem and long entity-distance problem. (iii) we conduct experiment on NYT dataset and achieve the state-of-the-art result.