当前位置: X-MOL 学术arXiv.cs.CL › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
arXiv - CS - Computation and Language Pub Date : 2020-03-22 , DOI: arxiv-2003.09833
Xiaoya Li, Yuxian Meng, Mingxin Zhou, Qinghong Han, Fei Wu and Jiwei Li

While the self-attention mechanism has been widely used in a wide variety of tasks, it has the unfortunate property of a quadratic cost with respect to the input length, which makes it difficult to deal with long inputs. In this paper, we present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection (SAC). In SAC, we regard the input sequence as a graph and attention operations are performed between linked nodes. In contrast with previous self-attention models with pre-defined structures (edges), the model learns to construct attention edges to improve task-specific performances. In this way, the model is able to select the most salient nodes and reduce the quadratic complexity regardless of the sequence length. Based on SAC, we show that previous variants of self-attention models are its special cases. Through extensive experiments on neural machine translation, language modeling, graph representation learning and image classification, we demonstrate SAC is competitive with state-of-the-art models while significantly reducing memory cost.

中文翻译:

SAC:通过稀疏自适应连接加速和构建自注意力

尽管自注意力机制已广泛应用于各种任务,但不幸的是,它具有相对于输入长度的二次成本的特性,这使得处理长输入变得困难。在本文中,我们提出了一种加速和构建自注意力的方法:稀疏自适应连接(SAC)。在 SAC 中,我们将输入序列视为一个图,并且在链接节点之间执行注意操作。与之前具有预定义结构(边)的自注意力模型相比,该模型学习构建注意力边以提高特定任务的性能。这样,无论序列长度如何,模型都能够选择最显着的节点并降低二次复杂度。基于 SAC,我们展示了之前的自注意力模型变体是它的特例。
更新日期:2020-09-30
down
wechat
bug