Multi-level graph neural network for text sentiment analysis
Graphical abstract
The multi-level graph neural network (MLGNN) considers both local features and global features, it applies node connection windows with different sizes and different message passing mechanisms at different levels. The bottom level involves a small connection window, which mainly focuses on the local features; the middle level has a larger connection window, mainly focusing on the long-distance features; the top level fully connects all the word nodes directly, which focuses on the global features.
Introduction
With the development of social networks, people surf the Internet and express their personal opinions on websites with increasing frequency [1]. The masses of text, including private emotional disposition, are a high-quality data source for data mining, especially by using sentiment analysis of text. According to the sentiment analysis results, different people often have different views and emotional tendencies toward the same thing, and by understanding this phenomenon, powerful functionality for competitive analysis and marketing analysis can be provided. Nowadays, sentiment analysis has been widely used in many fields, such as e-commerce, movie recommendations, and public opinion analysis [2]. For example, the text sentiment analysis of users’ evaluation of commodities on an e-commerce platform can provide valuable information in establishing a model of the user-commodity relationship. The e-commerce platform can provide personalized recommendations for users, and merchants can update their commodities according to the user-commodity relationship model in a timely manner. Text sentiment analysis of book reviews can help recommend books according to the reader's personal preferences. Publishers can better understand what books are popular based on the sentiment analysis result of book reviews, and then publish a related book that meets the public interest.
Text sentiment analysis, also known as opinion mining, is essentially a text classification task. Text sentiment analysis aims to classify text into a predefined sentiment polarity according to its content. Text sentiment analysis methods can mainly be classified into one of the following three groups: lexicon-based methods, shallow learning methods, and deep learning methods.
The lexicon-based methods mainly evaluate the texts with the emotional disposition by matching the sentiment words in the sentiment lexicon. The shallow learning methods focus on feature engineering, and they usually represent text with hand-crafted features, such as sparse lexical features. The lexical features mainly include bag-of-words (BOW) and n-grams. Classifiers are used in shallow learning methods. However, due to their limited expressive ability, it is difficult to improve the performance of these methods, and their generalization ability is also poor.
Deep learning methods have been widely applied in text sentiment analysis because word2vec [3] provides an effective way to learn distributed representations of words. The most popular deep neural networks used for text sentiment analysis are convolutional neural network (CNN) and recurrent neural network (RNN). In a CNN [4], the text features representation is constructed by acquiring the local information in the filter region, which makes it difficult for a CNN to learn the dependencies between distant positions. In comparison, RNN and its variants (long short-term memory (LSTM) [5] and gated recurrent unit (GRU) [6] can connect contextual memory and store more long-term global information, thus achieving the sentiment analysis of text. These methods mainly focus on consecutive word sequences, but they do not explicitly use word co-occurrence information, and the complex model structure is like a black-box.
In recent years, a new neural network, named graph neural network (GNN) [7], has aroused wide interest. In a GNN, a node collects information from other nodes and updates its representations. Prior work [8] has proposed using a graph convolutional network (GCN) [9] for text classification (Text_GCN). The Text_GCN model builds a graph for the whole corpus, which does not support online testing. Another study [10] proposed a text-level GNN for text classification, which can avoid graph construction for the entire corpus, and graphs are constructed for each input text. Like other GNN-based methods, this method [10] also only considers the adjacent nodes when updating the node representations of the graph, and it fails to consider the relationship between non-adjacent nodes. This causes the model to only focus on local features as in a CNN.
To deal with the problems above, we propose a novel multi-level graph neural network (MLGNN) for text sentiment analysis. Instead of building a graph for the entire corpus, we produce a graph for each input text. To imitate humans’ ability to understand language at different levels, the word nodes within a small window are connected at the bottom level, the word nodes within a larger window are connected at the middle level, finally, all the word nodes of the document are connected at the top level. The main contributions can be summarized as follows:
- 1
We integrate a scaled dot-product attention [11] mechanism into MLGNN for fusing the features of each word node in the graph.
- 2
To enable the model to focus on both local features and global features, we propose using connection windows with different sizes at different levels. In previous studies, the connection windows of the nodes were fixed.
- 3
The experimental results on public datasets demonstrated that, compared with similar methods, MLGNN could better handle text sentiment analysis tasks. At the same time, the influencing factors of MLGNN are also analyzed and discussed in this work.
The remainder of this paper is organized as follows: In Section 2, we review several related methods about text sentiment analysis. In Section 3, the MLGNN is introduced in detail. Section 4 verifies the effectiveness of MLGNN through experiments by comparing it with previous methods. Finally, a conclusion is made in Section 5.
Section snippets
Related works
At present, text sentiment analysis methods can be mainly divided into three categories: lexicon-based methods, shallow learning methods, and deep learning methods. In this section, we will review these methods and discuss their advantages and disadvantages.
Proposed approaches
In this section, we present the model structure of MLGNN in detail. First, we show how to build the graph and describe the message passing mechanism at the bottom level. Second, we introduce how to construct the graph structure and use the graph attention network (GAT) [26] to update node representations at the middle level. Then, we demonstrate how to update the node representations at the top level using the scaled dot-product attention mechanism. Finally, the output and loss functions of the
Experiments
In this section, we describe our experimental setup and report our experimental results.
Conclusion and future work
Sentiment analysis is one of the research fields of NLP. This paper proposes a GNN model MLGNN with different sizes of connection windows at different levels, in which the node representations are updated with different message passing mechanisms. Specifically, we propose to use a small connection window at the bottom layer and aggregate the feature representations of adjacent words by average. In the middle layer, we use a larger connection window, and GAT is applied as the message passing
Declaration of Competing Interest
The authors declared that they have no conflicts of interest to this work.
Author statement
The research in this paper is carried out under the supervision of Bi Zeng and Jianqi Liu. All authors contributed as follows:
Wenxiong Liao: Performing the main experiments and writing-original manuscript.
Bi Zeng: Theoretical guidance and experiments guidance.
Jianqi Liu: Guidance on paper revision, fund support and as the corresponding author of this article.
Pengfei Wei: Experimental programming and data processing.
Xiaochun Cheng: Writing-reviewing and editing.
Weiwen Zhang: Visualization and
Acknowledgment
This work was partially supported by the Science and Technology Program of Guangzhou, China (No. 201804010238), the Natural Science Foundations of Guangdong Province, China (2018A030310540), Guangdong Basic and Applied Basic Research Foundation (No. 2019A1515011056), the National Natural Science Foundation of China (No. 61701122). .
WenXiong Liao was born in 1995. He received the bachelor's degree in Information Management and Information System from Guangzhou Medical University (GZMU). He received the degree of M.Eng. in computer technology from Guangdong University of Technology (GDUT), China. His-current research interests include AI and natural language processing.
References (30)
- et al.
Social image sentiment analysis by exploiting multimodal content and heterogeneous relations
IEEE Trans Ind Inf
(2020) - et al.
A review of emotion sensing: categorization models and algorithms
(2020) - et al.
Interpreting the public sentiment variations on twitter
IEEE Trans Knowl Data Eng
(2014) - et al.
Distributed representations of words and phrases and their compositionality
Adv Neural Inf Process Syst
(2013) - Y. Kim. Convolutional neural networks for sentence classification. arXiv preprint...
- et al.
Long Short-Term Memory
Neural Comput
(1997) - J. Chung, C. Gulcehre, K. Cho, Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling....
- et al.
The graph neural network model
IEEE Trans Neural Netw
(2008) - et al.
Graph convolutional networks for text classification
- T.N. Kipf, M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint...
Text level graph neural network for text classification
Attention is all you need
Adv Neural Inf Process Syst
Sentiwordnet: a publicly available lexical resource for opinion mining
LREC Citeseer
Performance enhancement in P300 ERP single trial by machine learning adaptive denoising mechanism
IEEE Network Lett
Optimal feature selection for learning-based algorithms for sentiment classification
Cognit Comput
Cited by (45)
Boosting short term electric load forecasting of high & medium voltage substations with visibility graphs and graph neural networks
2024, Sustainable Energy, Grids and NetworksLearning interactions across sentiment and emotion with graph attention network and position encodings
2024, Pattern Recognition LettersNetwork traffic prediction with Attention-based Spatial–Temporal Graph Network
2024, Computer NetworksIntegrating sentiment analysis with graph neural networks for enhanced stock prediction: A comprehensive survey
2024, Decision Analytics JournalFeature interactive graph neural network for KG-based recommendation[Formula presented]
2024, Expert Systems with ApplicationsDCENet: A dynamic correlation evolve network for short-term traffic prediction
2023, Physica A: Statistical Mechanics and its Applications
WenXiong Liao was born in 1995. He received the bachelor's degree in Information Management and Information System from Guangzhou Medical University (GZMU). He received the degree of M.Eng. in computer technology from Guangdong University of Technology (GDUT), China. His-current research interests include AI and natural language processing.
Bi Zeng obtained both her degrees of M.S. and Ph.D. from Guangdong University of Technology (GDUT). Now, she is a Professor in the School of Computers, GDUT. At present, her researches focus on the areas of intelligent robot, computational intelligence, data mining, and wireless sensor networks.
Jianqi Liu received his Ph.D. degree in control science and engineering from the School of Automation, Guangdong University of Technology (GDUT) in 2016. He is an Associate Professor in the School of Automation, GDUT, China. His-has broad research interests, including big data, the Internet of Vehicles (IoV), and the cyber-physical systems. He is a member of IEEE.
Pengfei Wei was born in 1991. He received his B. E. degree from Zhengzhou University, China. Now, he is studying for a M.Eng. degree in Guangdong University of Technology (GDUT), China. His-current research interests include natural language processing, task dialog system, reinforcement learning, graph neural network.
Xiaochun Cheng (SM’04) received his Ph.D. in Computer Science in 1996, and a degree of Executive MBA in 2011. Since 2012, he has worked as the Computer Science Project Coordinator in Middlesex University. He is the member of the IEEE SMC Technical Committee on Computational Intelligence, the IEEE Communications Society Communications and Information Security Technical Committee.
Weiwen Zhang received the Ph.D. in Computer Engineering from Nanyang Technological University (NTU), Singapore in 2015. Now, he is an Associate Professor in the School of Computers at Guangdong University of Technology (GDUT), China. His-research interests include the areas of cloud computing, mobile computing, big data analytics and machine learning. He is an IEEE member.