S2SAN: A sentence-to-sentence attention network for sentiment analysis of online reviews

Highlights

• Introduce a sentence-level attention mechanism “sentence-to-sentence attention”.

• Proposed S2SAN outperforms baselines in domain-specific, cross- and multi-domain sentiment analysis tasks.

• Some classifiers yield better accuracy when embedded into a sentence-to-sentence attention framework.

Abstract

Many existing attention-based deep learning approaches to sentiment analysis have focused on words and represent an entire review text as a word sequence. However, these approaches overlook the differences in the importance of each sentence to the complete text. To solve this problem, some work has been performed to calculate sentence-level attention, but these studies use the same approach that is applied to word-level attention, which leads to unnecessary sequential structures and increased complexity of sentence representation. Therefore, in this paper, we propose a sentence-to-sentence attention network¹ (S2SAN) using multihead self-attention. We conducted several domain-specific, cross-domain and multidomain sentiment analysis experiments with real-world datasets. The experimental results show that S2SAN outperforms other state-of-the-art models. Some classical sentiment classifiers [e.g., convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM) models] achieve better accuracies when they are reconfigured to include sentence-to-sentence attention.

Introduction

The rise of the e-commerce industry has caused substantial changes in business models between merchants and consumers, which has enabled exponential proliferation in the number of online reviews [[1], [2], [3]]. Websites such as Amazon and Jingdong provide customers with a platform for sharing their perspectives and opinions on various products and services [4]. Rich sentiment information exists in online reviews that directly reflect the positive, negative or neutral sentiment polarity of users [5,6]. Note that reviews of products and services in online platforms offer value from a variety of perspectives. The ‘Electronic Word of Mouth’, which is a collection of reviews, is a key driver in decision-making [7,8]. On the other hand, reviews offer clues to users' latent demands and motivations, which render them conducive for improving online merchants' products and service quality [9,10]. Moreover, the potential user interests and preferences revealed in reviews provide reference information for making personalized recommendations [11]. However, the vast number of online reviews hinder the ability of both users and merchants to comprehensively extract and analyze public opinions [12]. To fully utilize the immense value of online reviews, there is a pressing need for a sentiment analysis model that can convert the overloaded sentiment information contained in reviews into a more easily understandable form.

Sentiment analysis, which is also known as opinion mining [13], is dedicated to classifying the sentiment polarity in individual review texts in our work. Previous approaches to sentiment analysis include labeling corpora that are intended for sentiment classification tasks [14,15], building sentiment lexicons [16,17], and training domain-specific sentiment classifiers [15,18,19], all of which have laid a solid foundation for sentiment analysis research. With the increasing computational ability of computers and the continual development of deep learning technology, methods that combine language models and neural networks have become mainstream approaches for solving sentiment analysis problems. These approaches are superior to traditional machine learning classifier models in terms of text representation and deep semantic understanding [12,20,21] and have greatly improved the sentiment analysis performance.

In recent years, attention-based neural network models have achieved remarkable results in many natural language processing (NLP) tasks. Additionally, a large number of studies have applied attention mechanisms to sentiment analysis. The attention mechanism in deep learning is a simulation of the visual attention of human beings [22]. Human beings always ignore information irrelevant to the target task and allocate more attention resources to important information. The attention mechanism helps improve the robustness of deep learning models by extracting useful information from a large number of inputs and allocating more attention weights to this useful information. In this regard, attention mechanisms have made satisfactory progress. Generally, attention mechanism can tell how important every entity is in an entity sequence. An illustration of the case where the entities are words is presented in Section 3.2.1.

However, to the best of our knowledge, existing attention-based neural network models focus on words [[23], [24], [25]] by representing the entire text as a word sequence. The attention weights of different words are calculated based on their similarity in a word attention layer. However, in our research, we suggest that each sentence in a text expresses a unique semantic meaning; thus, the attention weights of different sentences should be calculated individually. Although some works (such as the hierarchical attention network (HAN) proposed by Yang et al. [26]) consider sentence-level attention, most studies calculate sentence-level attention in the same way as word-level attention. However, we suggest that the approaches that directly transplant word-encoding methods into sentence-encoding have a defect: no apparent sequence-structure relationship exists between the sentences in a review, as a result, representing sentences in a review using a sequence model adds unnecessary complexity. To fill this gap, we introduce a novel sentence-to-sentence structure where self-attention is used to calculate relation weights of every two sentences. Compared with the original attention mechanism, self-attention can provide a matrix where the relations of any two entities are represented. An illustration of the case where the entities are sentences is presented in Section 3.2.2.

Our research objectives are three-fold. The first objective is to design the sentence-to-sentence attention network (S2SAN). In S2SAN, the most suitable word encoder and sentence encoder for online review texts should be figured out. The second objective is to certify whether the S2SAN is superior to the sequence of sentences network (HAN) in terms of the classification accuracy and model training time in domain-specific, cross-domain, multidomain sentiment analysis tasks. We suggest that the proposed sentence-to-sentence attention model has excellent compatibility with different word encoders. Therefore, our third objective is to verify whether some extensively applied word encoders can achieve better accuracies with the reconfiguration of sentence-to-sentence attention.

Overall, our contributions are summarized as follows:

• To the best of our knowledge, this is the first method that uses sentence-to-sentence attention to realize the sentence-level attention mechanism. This approach disregards sentence positional information and reduces the complexity involved in building sentence sequences.

• Aimed at online reviews, S2SAN achieves excellent results in domain-specific, cross-domain and multidomain sentiment analysis experiments. The proposed model increases accuracy by 0.8% to 2.1% and reduces the training time by an average of 25% compared to another hierarchical model (HAN).

• In several parallel experiments, we verify that after optimization with a hierarchical structure and the sentence-to-sentence attention framework, the accuracies of some classical sentiment classifiers (e.g., convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM) models) increase by 1.9% to 8%.

The remainder of this paper is organized as follows: Section 2 outlines the most relevant related research regarding sentiment analysis of online reviews and attention mechanisms. The sentence-to-sentence attention model is presented in Section 3. Section 4 reports the experimental results, which are derived from multiple review datasets. The experimental results are discussed in Section 5. Section 6 provides the conclusions, limitations, and suggested directions for future work.