当前位置: X-MOL 学术arXiv.cs.CV › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
arXiv - CS - Computer Vision and Pattern Recognition Pub Date : 2021-05-05 , DOI: arxiv-2105.02358
Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, Shi-Min Hu

Attention mechanisms, especially self-attention, play an increasingly important role in deep feature representation in visual tasks. Self-attention updates the feature at each position by computing a weighted sum of features using pair-wise affinities across all positions to capture long-range dependency within a single sample. However, self-attention has a quadratic complexity and ignores potential correlation between different samples. This paper proposes a novel attention mechanism which we call external attention, based on two external, small, learnable, and shared memories, which can be implemented easily by simply using two cascaded linear layers and two normalization layers; it conveniently replaces self-attention in existing popular architectures. External attention has linear complexity and implicitly considers the correlations between all samples. Extensive experiments on image classification, semantic segmentation, image generation, point cloud classification and point cloud segmentation tasks reveal that our method provides comparable or superior performance to the self-attention mechanism and some of its variants, with much lower computational and memory costs.

中文翻译:

超越自我注意力:视觉任务使用两个线性层的外部注意力

注意机制,特别是自我注意,在视觉任务的深度特征表示中起着越来越重要的作用。自我关注通过使用所有位置之间的成对亲和力来计算特征的加权总和,以捕获单个样本内的长期依赖性,从而更新每个位置的特征。但是,自我注意具有二次复杂度,并且忽略了不同样本之间的潜在相关性。本文基于两个外部的,小的,可学习的和共享的存储器,提出了一种称为外部注意力的新颖注意力机制,只需使用两个层叠的线性层和两个归一化层即可轻松实现。它可以方便地取代现有流行架构中的注意力。外部注意力具有线性复杂度,并且隐式考虑了所有样本之间的相关性。在图像分类,语义分割,图像生成,点云分类和点云分割任务方面的大量实验表明,我们的方法可以提供与自注意力机制及其某些变体相当或更高的性能,而计算和内存成本要低得多。
更新日期:2021-05-07
down
wechat
bug