Multi-level graph neural network for text sentiment analysis

https://doi.org/10.1016/j.compeleceng.2021.107096Get rights and content

Highlights

  • The previous graph neural networks used for text sentiment analysis cannot consider both local and global features.

  • A multi-level graph neural network for text sentiment analysis was proposed.

  • Use different edge connection methods and different messaging mechanisms at different levels.

  • Compared with previous methods, our method can better handle text sentiment analysis task.

Abstract

Text sentiment analysis is a fundamental task in the field of natural language processing (NLP). Recently, graph neural networks (GNNs) have achieved excellent performance in various NLP tasks. However, a GNN only considers the adjacent words when updating the node representations of the graph, and thus the model can only focus on the local features while ignoring global features. In this paper, we propose a novel multi-level graph neural network (MLGNN) for text sentiment analysis. To consider both local features and global features, we apply node connection windows with different sizes at different levels. Particularly, we integrate a scaled dot-product attention mechanism as a message passing mechanism into our method for fusing the features of each word node in the graph. The experimental results demonstrated that the proposed model outperformed other models in text sentiment analysis tasks.

Graphical abstract

The multi-level graph neural network (MLGNN) considers both local features and global features, it applies node connection windows with different sizes and different message passing mechanisms at different levels. The bottom level involves a small connection window, which mainly focuses on the local features; the middle level has a larger connection window, mainly focusing on the long-distance features; the top level fully connects all the word nodes directly, which focuses on the global features.

Image, graphical abstract
  1. Download : Download high-res image (130KB)
  2. Download : Download full-size image

Introduction

With the development of social networks, people surf the Internet and express their personal opinions on websites with increasing frequency [1]. The masses of text, including private emotional disposition, are a high-quality data source for data mining, especially by using sentiment analysis of text. According to the sentiment analysis results, different people often have different views and emotional tendencies toward the same thing, and by understanding this phenomenon, powerful functionality for competitive analysis and marketing analysis can be provided. Nowadays, sentiment analysis has been widely used in many fields, such as e-commerce, movie recommendations, and public opinion analysis [2]. For example, the text sentiment analysis of users’ evaluation of commodities on an e-commerce platform can provide valuable information in establishing a model of the user-commodity relationship. The e-commerce platform can provide personalized recommendations for users, and merchants can update their commodities according to the user-commodity relationship model in a timely manner. Text sentiment analysis of book reviews can help recommend books according to the reader's personal preferences. Publishers can better understand what books are popular based on the sentiment analysis result of book reviews, and then publish a related book that meets the public interest.

Text sentiment analysis, also known as opinion mining, is essentially a text classification task. Text sentiment analysis aims to classify text into a predefined sentiment polarity according to its content. Text sentiment analysis methods can mainly be classified into one of the following three groups: lexicon-based methods, shallow learning methods, and deep learning methods.

The lexicon-based methods mainly evaluate the texts with the emotional disposition by matching the sentiment words in the sentiment lexicon. The shallow learning methods focus on feature engineering, and they usually represent text with hand-crafted features, such as sparse lexical features. The lexical features mainly include bag-of-words (BOW) and n-grams. Classifiers are used in shallow learning methods. However, due to their limited expressive ability, it is difficult to improve the performance of these methods, and their generalization ability is also poor.

Deep learning methods have been widely applied in text sentiment analysis because word2vec [3] provides an effective way to learn distributed representations of words. The most popular deep neural networks used for text sentiment analysis are convolutional neural network (CNN) and recurrent neural network (RNN). In a CNN [4], the text features representation is constructed by acquiring the local information in the filter region, which makes it difficult for a CNN to learn the dependencies between distant positions. In comparison, RNN and its variants (long short-term memory (LSTM) [5] and gated recurrent unit (GRU) [6] can connect contextual memory and store more long-term global information, thus achieving the sentiment analysis of text. These methods mainly focus on consecutive word sequences, but they do not explicitly use word co-occurrence information, and the complex model structure is like a black-box.

In recent years, a new neural network, named graph neural network (GNN) [7], has aroused wide interest. In a GNN, a node collects information from other nodes and updates its representations. Prior work [8] has proposed using a graph convolutional network (GCN) [9] for text classification (Text_GCN). The Text_GCN model builds a graph for the whole corpus, which does not support online testing. Another study [10] proposed a text-level GNN for text classification, which can avoid graph construction for the entire corpus, and graphs are constructed for each input text. Like other GNN-based methods, this method [10] also only considers the adjacent nodes when updating the node representations of the graph, and it fails to consider the relationship between non-adjacent nodes. This causes the model to only focus on local features as in a CNN.

To deal with the problems above, we propose a novel multi-level graph neural network (MLGNN) for text sentiment analysis. Instead of building a graph for the entire corpus, we produce a graph for each input text. To imitate humans’ ability to understand language at different levels, the word nodes within a small window are connected at the bottom level, the word nodes within a larger window are connected at the middle level, finally, all the word nodes of the document are connected at the top level. The main contributions can be summarized as follows:

  • 1

    We integrate a scaled dot-product attention [11] mechanism into MLGNN for fusing the features of each word node in the graph.

  • 2

    To enable the model to focus on both local features and global features, we propose using connection windows with different sizes at different levels. In previous studies, the connection windows of the nodes were fixed.

  • 3

    The experimental results on public datasets demonstrated that, compared with similar methods, MLGNN could better handle text sentiment analysis tasks. At the same time, the influencing factors of MLGNN are also analyzed and discussed in this work.

The remainder of this paper is organized as follows: In Section 2, we review several related methods about text sentiment analysis. In Section 3, the MLGNN is introduced in detail. Section 4 verifies the effectiveness of MLGNN through experiments by comparing it with previous methods. Finally, a conclusion is made in Section 5.

Section snippets

Related works

At present, text sentiment analysis methods can be mainly divided into three categories: lexicon-based methods, shallow learning methods, and deep learning methods. In this section, we will review these methods and discuss their advantages and disadvantages.

Proposed approaches

In this section, we present the model structure of MLGNN in detail. First, we show how to build the graph and describe the message passing mechanism at the bottom level. Second, we introduce how to construct the graph structure and use the graph attention network (GAT) [26] to update node representations at the middle level. Then, we demonstrate how to update the node representations at the top level using the scaled dot-product attention mechanism. Finally, the output and loss functions of the

Experiments

In this section, we describe our experimental setup and report our experimental results.

Conclusion and future work

Sentiment analysis is one of the research fields of NLP. This paper proposes a GNN model MLGNN with different sizes of connection windows at different levels, in which the node representations are updated with different message passing mechanisms. Specifically, we propose to use a small connection window at the bottom layer and aggregate the feature representations of adjacent words by average. In the middle layer, we use a larger connection window, and GAT is applied as the message passing

Declaration of Competing Interest

The authors declared that they have no conflicts of interest to this work.

Author statement

The research in this paper is carried out under the supervision of Bi Zeng and Jianqi Liu. All authors contributed as follows:

Wenxiong Liao: Performing the main experiments and writing-original manuscript.

Bi Zeng: Theoretical guidance and experiments guidance.

Jianqi Liu: Guidance on paper revision, fund support and as the corresponding author of this article.

Pengfei Wei: Experimental programming and data processing.

Xiaochun Cheng: Writing-reviewing and editing.

Weiwen Zhang: Visualization and

Acknowledgment

This work was partially supported by the Science and Technology Program of Guangzhou, China (No. 201804010238), the Natural Science Foundations of Guangdong Province, China (2018A030310540), Guangdong Basic and Applied Basic Research Foundation (No. 2019A1515011056), the National Natural Science Foundation of China (No. 61701122). .

WenXiong Liao was born in 1995. He received the bachelor's degree in Information Management and Information System from Guangzhou Medical University (GZMU). He received the degree of M.Eng. in computer technology from Guangdong University of Technology (GDUT), China. His-current research interests include AI and natural language processing.

References (30)

  • J. Xu et al.

    Social image sentiment analysis by exploiting multimodal content and heterogeneous relations

    IEEE Trans Ind Inf

    (2020)
  • Z. Wang et al.

    A review of emotion sensing: categorization models and algorithms

    (2020)
  • S. Tan et al.

    Interpreting the public sentiment variations on twitter

    IEEE Trans Knowl Data Eng

    (2014)
  • T. Mikolov et al.

    Distributed representations of words and phrases and their compositionality

    Adv Neural Inf Process Syst

    (2013)
  • Y. Kim. Convolutional neural networks for sentence classification. arXiv preprint...
  • S. Hochreiter et al.

    Long Short-Term Memory

    Neural Comput

    (1997)
  • J. Chung, C. Gulcehre, K. Cho, Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling....
  • F. Scarselli et al.

    The graph neural network model

    IEEE Trans Neural Netw

    (2008)
  • L. Yao et al.

    Graph convolutional networks for text classification

  • T.N. Kipf, M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint...
  • L. Huang et al.

    Text level graph neural network for text classification

  • A. Vaswani et al.

    Attention is all you need

    Adv Neural Inf Process Syst

    (2017)
  • A. Esuli et al.

    Sentiwordnet: a publicly available lexical resource for opinion mining

    LREC Citeseer

    (2006)
  • S.K. Haider et al.

    Performance enhancement in P300 ERP single trial by machine learning adaptive denoising mechanism

    IEEE Network Lett

    (2018)
  • Z. Wang et al.

    Optimal feature selection for learning-based algorithms for sentiment classification

    Cognit Comput

    (2020)
  • Cited by (45)

    • DCENet: A dynamic correlation evolve network for short-term traffic prediction

      2023, Physica A: Statistical Mechanics and its Applications
    View all citing articles on Scopus

    WenXiong Liao was born in 1995. He received the bachelor's degree in Information Management and Information System from Guangzhou Medical University (GZMU). He received the degree of M.Eng. in computer technology from Guangdong University of Technology (GDUT), China. His-current research interests include AI and natural language processing.

    Bi Zeng obtained both her degrees of M.S. and Ph.D. from Guangdong University of Technology (GDUT). Now, she is a Professor in the School of Computers, GDUT. At present, her researches focus on the areas of intelligent robot, computational intelligence, data mining, and wireless sensor networks.

    Jianqi Liu received his Ph.D. degree in control science and engineering from the School of Automation, Guangdong University of Technology (GDUT) in 2016. He is an Associate Professor in the School of Automation, GDUT, China. His-has broad research interests, including big data, the Internet of Vehicles (IoV), and the cyber-physical systems. He is a member of IEEE.

    Pengfei Wei was born in 1991. He received his B. E. degree from Zhengzhou University, China. Now, he is studying for a M.Eng. degree in Guangdong University of Technology (GDUT), China. His-current research interests include natural language processing, task dialog system, reinforcement learning, graph neural network.

    Xiaochun Cheng (SM’04) received his Ph.D. in Computer Science in 1996, and a degree of Executive MBA in 2011. Since 2012, he has worked as the Computer Science Project Coordinator in Middlesex University. He is the member of the IEEE SMC Technical Committee on Computational Intelligence, the IEEE Communications Society Communications and Information Security Technical Committee.

    Weiwen Zhang received the Ph.D. in Computer Engineering from Nanyang Technological University (NTU), Singapore in 2015. Now, he is an Associate Professor in the School of Computers at Guangdong University of Technology (GDUT), China. His-research interests include the areas of cloud computing, mobile computing, big data analytics and machine learning. He is an IEEE member.

    View full text