Self attention mechanism is widely used in relation extraction, emotion classification and other tasks. It can extract a wide range of relevance information in the text. The attention mode of the existing self attention mechanism is soft attention mode, that is, a dense attention matrix is generated by softmax function. However, if the sentence length is long, the weight of important information will be too small. At the same time, the softmax function assumes that all elements have a positive impact on the results by default, which makes the model unable to extract the negative effect information. We use hard attention mechanism, namely sparse attention matrix, to improve the existing self attention model and fully extract the positive and negative information of text. Our model can not only enhance the extraction of positive information, but also makes up for the blank that the traditional attention matrix cannot be negative. We evaluated our model in three tasks and seven data sets. The experimental results show that our model is superior to the traditional self attention model and superior to state-of-the-art models in some tasks.

自注意力机制广泛应用于关系抽取、情感分类等任务。它可以提取文本中广泛的相关信息。现有的自注意力机制的注意力模式是软注意力模式，即通过softmax函数生成一个密集的注意力矩阵。但是，如果句子长度很长，重要信息的权重就会太小。同时，softmax 函数默认假设所有元素对结果都有正面影响，这使得模型无法提取负面影响信息。我们使用硬注意力机制，即稀疏注意力矩阵，来改进现有的自我注意力模型，充分提取文本的正负信息。我们的模型不仅可以增强正面信息的提取，也弥补了传统注意力矩阵不能为负的空白。我们在三个任务和七个数据集中评估了我们的模型。实验结果表明，我们的模型优于传统的自我注意模型，并且在某些任务上优于最先进的模型。