GAFM: A Knowledge Graph Completion Method Based on Graph Attention Faded Mechanism

https://doi.org/10.1016/j.ipm.2022.103004Get rights and content

Highlights

  • Attention mechanism helps to obtain information in the neighborhood.

  • Incorporate the information in multi-hop neighborhood to embed the target entitie.

  • Adjust the attention value according to the variation of path length.

  • Graph Attention Faded Mechiasm can help to obtain more appropriate attention value.

Abstract

Although the Knowledge Graph (KG) has been successfully applied to various applications, there is still a large amount of incomplete knowledge in the KG. This study proposes a Knowledge Graph Completion (KGC) method based on the Graph Attention Faded Mechanism (GAFM) to solve the problem of incomplete knowledge in KG. GAFM introduces a graph attention network that incorporates the information in multi-hop neighborhood nodes to embed the target entities into low dimensional space. To generate a more expressive entity representation, GAFM gives different weights to the neighborhood nodes of the target entity by adjusting the attention value of neighborhood nodes according to the variation of the path length. The attention value is adjusted by the attention faded coefficient, which decreases with the increase of the distance between the neighborhood node and the target entity. Then, considering that the capsule network has the ability to fit features, GAFM introduces the capsule network as the decoder to extract feature information from triple representations. To verify the effectiveness of the proposed method, we conduct a series of comparative experiments on public datasets (WN18RR and FB15k-237). Experimental results show that the proposed method outperforms baseline methods. The Hits@10 metric is improved by 8% compared with the second-place KBGAT method.

Introduction

Knowledge Graph (KG) is a large network of entities, their semantic types, properties, and relations between entities (Ehrlinger & Wöß, 2016). It aims to describe the association between entities or concepts. KG represents a piece of knowledge in the form of “node-edge-node”. As one of the triples in Fig. 1 (Jonathan Nolan; born in; London), the nodes “Jonathan Nolan” and “London” correspond to two entities, and the edge “born in” reflects the association between these two entities. By doing so, the knowledge consisting of various entities, concepts, and relations is stored and presented in the KG. Specifically, knowledge graphs (KGs) have made a significant contribution to the field of Natural Language Processing (NLP). Such as Named-entity Recognition (He, Wu, Yin, Cai, & Assoc Advancement Artificial, 2020) and Entity Disambiguation (Feng, Wang, Jiang, Lyu, & Zhu, 2020; Mulang et al., 2020). Besides that, KGs also provide powerful support for Artificial Intelligence applications and have been well used in the field of cultural protection (Fan & Wang, 2022), question answering system (Qiao & Hu, 2020), smart healthcare (Yu, Wang, & Zhang, 2021; Zhang et al., 2020), social network analysis (Qian et al., 2019), and recommendation systems (Liu et al., 2021; X. Wang et al., 2019) successfully.

Despite the fact that KGs have been applied to a wide range of fields, they still include a significant amount of incomplete knowledge, i.e., incomplete triples with missing entities or relations. For example, large KGs like YAGO (Suchanek, Kasneci, & Weikum, 2007) and DBPedia (Lehmann et al., 2015), which contain billions of facts, also inevitably have knowledge integrity problems (Krompaß, Baier, & Tresp, 2015). Therefore, the Knowledge Graph Completion (KGC) technology is particularly valuable in dealing with problems of incomplete knowledge. Because most existing KG needs to be completed to improve its quality continuously, researchers have been interested in KGC technologies. The goal of this paper is to achieve better completion effect than the existing KGC methods, so that the triples in the KG can represent knowledge more completely.

In the field of KGC, the most popular method is representation learning. This kind of method not only has strong generalization ability but also has good scalability. Representation learning has produced noteworthy achievements on KGC along with the development of representation learning (Jagvaral, Lee, Roh, Kim, & Park, 2020; Zhang et al., 2020). Representation learning is used to automatically mine the implicit features of data, and it projects entities or relations into a low-dimensional vector space. Eventually, the semantic information of entities and relations is expressed in symbolic values. Representation learning has a number of advantages, including effectively simplifying complex input data (Zhang et al., 2021).

Among existing representation learning models, some of them, for example, the translation model (e.g., TransE (Bordes, Usunier, Garcia-Duran, Weston, & Yakhnenko, 2013)), the neural network models (e.g., ConvE (Dettmers, Minervini, Stenetorp, & Riedel, 2018) and ConvKB (Dai Quoc, Dat Quoc, Tu Dinh, & Dinh, 2019)) are only capable of dealing with individual triples as they do not consider the neighborhood information of entities. Therefore, the Graph Convolutional Network (GCN) (Kipf & Welling, 2016) is proposed to exploit the feature information of a node's first-order neighbours. On its basis, the Graph Attention Network (GAT) (Veličković et al., 2017) combines the attention mechanism to achieve further improvements in the completion effect. However, GAT (Veličković et al., 2017) does not consider the contribution of relational features to entity embedding. This leaves relations, one of the key elements of knowledge, without being utilized. KBGAT (Nathani, Chauhan, Sharma, Kaul, & Acl, 2019), which is based on GAT (Veličković et al., 2017), further introduces node information in multi-hop neighborhoods to deal with this problem. The contribution of relations is finally taken into account by KBGAT (Nathani et al., 2019), which extends the attention mechanism to the extraction of relational information.

Although the existing KGC methods have achieved some results, there is still room for improvement in the current KGC methods. On the one hand, some methods are only capable of dealing with a single triple and as they only use a small amount of information from the triple itself. On the basis of the above methods, some methods introduce the neural network model for feature extraction. Although these methods combine the neural network model to obtain the implicit features, as the triple does not contain sufficient information, these kinds of methods cannot achieve a good completion effect. On the other hand, the existing methods not fully utilized the neighborhood information of the target entity. In addition, some mainstream completion methods learn the triple representation automatically by leveraging deep learning techniques. However, because they give the same attention to the neighborhood nodes associated with the entities, the information contained in the network structure is not fully utilized in the process of triple representation learning.

In order to solve the problems of existing methods and further promote the improvement of the effect of the KGC method, this paper proposes a KGC method based on the Graph Attention Faded Mechanism (GAFM). Specifically, we first obtain the path information around the given node. Then, we introduce the graph attention faded mechanism to manage the quantity of information aggregated from the target entity's neighborhood nodes. Specifically, it combines the path length information according to the length of the path so that the attention value of the neighborhood information varies with the change of the length. The attention value is reduced when the neighborhood node is far away from the target entity. Thus, we obtain the neighborhood information of the target entity selectively. Therefore, GAFM achieves a better information control ability for the neighborhood information from the target entity. In addition, because CapsE (Vu, Nguyen, Nguyen, & Phung, 2019) can not only capture the characteristic information in the same dimension in the embedding matrix of triples, but also can analyze the global embedding features of triples in each dimension, and has good performance (Vu et al., 2019), this paper employs CapsE (Vu et al., 2019) as the decoder to score the candidate triples with fused path information and identify whether the candidate triples are valid or not. Next, select valid triples and add them to the KG. Ultimately, entity completion of the KG is achieved.

The main contribution of this paper is as follows:

  • (1)

    We propose an attention faded mechanism based on GAT (Veličković et al., 2017) to adjust the neighborhoods’ attention value according to the path length. When embedding the target entity, with the use of the attention mechanism, it can selectively capture information from its n-th order neighborhood, leading to a more expressive entity embedding.

  • (2)

    We encode the entity with the combination of the graph attention faded mechanism and employ the capsule network (CapsE (Vu et al., 2019)) to calculate the score of the candidate triples. The score is then used to judge the validity of the candidate triples.

  • (3)

    We conduct a series of KGC experiments on the public datasets (WN18RR, FB15k-237) to verify the effectiveness of the proposed method. The experimental results on FB15k-237 show that the Hits@3 and Hits@10 assessment metrics are improved by 7% and 8%, respectively, over the second-place baseline approach KBGAT (Nathani et al., 2019).

The remainder of this paper is organized as follows. The work related to KGC in two technological directions are presented in Section 2. Section 3 elaborates on the primary issues addressed in this paper. The proposed method is detailed in Section 4. Section 5 presents the experimental results and the result analysis. Section 6 concludes this paper and states the planned future works.

Section snippets

Related Work

Researchers have proposed various distinct representation learning methods in the field of KGC research, including translation model-based method and neural network-based method. Besides that, the combination of attention mechanism on neural network models has also received increasing attention.

Problem Statement

In this paper, KG represents knowledge using RDF (Resource Description Framework) triples in the form of (head entity(h), relation(r), tail entity(t)). Each piece of knowledge is represented with two nodes indicating the head entity (h), tail entity (t), and the relation (r) between entities, which is marked on the edge connecting nodes as shown in Fig 1. The declaration of some symbols used in this paper is given in Table 2.

The purpose of KGC is to predict missing parts in triples so as to

Proposed Method

This section describes the proposed method in detail. Firstly, it introduces the graph attention network combining relational information, which improves and extends GAT (Veličković et al., 2017) network to incorporate relational information into embedding the target entities. Next, on this basis, the attention faded coefficient based on path length is proposed in this paper to adjust the attention value. Then, the CapsE (Vu et al., 2019) as the decoder was chosen to use. Fig. 2 depicts the

Dataset and experimental setup

In this paper, the GAFM's performance in the task of link prediction is utilized to judge its effectiveness. The link prediction task is faced with the case of missing the head or tail entities, i.e., (?,rk,ej)or (ei,rk,?). The execution process of link prediction task is as follows: first, the head or tail entities of the qualified triples are subjected to a deletion operation, and the missing parts are filled with other entities in the dataset, which is done here using Bernoulli sampling.

Conclusions

This paper proposes a method named GAFM for KGC. In the encoding layer, this paper proposes graph attention faded mechanism to improve the attention mechanism. It adjusts the attention value according to the change of the path length of given target entity. By doing so, the attention value fades with the increase of the distance from the neighborhood entity to the target entity so as to obtain an expressive entity representation. In the decoding layer, this paper introduces the CapsE model to

Acknowledgments

This work was supported by the Henan Province Science and Technology Department Foundation [No. 222102210027, 202102310295]; and the Doctoral Research Fund of Zhengzhou University of Light Industry [No. 2018BSJJ039]; and the National Nature Science Foundation of China [No. 61802352, 61802353, 61872278]; and the Postgraduate education reform and quality improvement project of Henan Province [No.YJS2021KC12]; and the Henan Province Science Foundation for Youths [No. 222300420230]; and the Open

References (29)

  • B. Jagvaral et al.

    Path-based reasoning approach for knowledge graph completion using CNN-BiLSTM with attention mechanism

    Expert Systems with Applications

    (2020)
  • A. Bordes et al.

    Translating embeddings for modeling multi-relational data

    (2013)
  • N. Dai Quoc et al.

    A convolutional neural network-based model for knowledge base completion and its application to search personalization

    Semantic Web

    (2019)
  • T. Dettmers et al.

    Convolutional 2d knowledge graph embeddings

  • L. Ehrlinger et al.

    Towards a definition of knowledge graphs

    SEMANTiCS

    (2016)
  • T. Fan et al.

    Research of Chinese intangible cultural heritage knowledge graph construction and attribute value extraction with graph attention network

    Information Processing & Management

    (2022)
  • Z. Feng et al.

    Knowledge-Enhanced Named Entity Disambiguation for Short Text

  • Q.Z. He et al.

    Feb 07-12). Knowledge-Graph Augmented Word Representations for Named Entity Recognition

  • G.L. Ji et al.

    Knowledge Graph Embedding via Dynamic Mapping Matrix

  • Kipf, T. N., & Welling, M. J. a. p. a. (2016). Semi-supervised classification with graph convolutional...
  • D. Krompaß et al.

    Type-Constrained Representation Learning in Knowledge Graphs

    Cham

    (2015)
  • J. Lehmann et al.

    DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia

    Semantic Web

    (2015)
  • Lin, D., Tang, J., Li, X., Pang, K., Li, S., & Wang, T. (2022). BERT-SMAP: Paying attention to Essential Terms in...
  • Y. Lin et al.

    Learning Entity and Relation Embeddings for Knowledge Graph Completion

  • Cited by (0)

    View full text