While attention has been an increasingly popular component in deep neural networks to both interpret and boost the performance of models, little work has examined how attention progresses to accomplish a task and whether it is reasonable. In this work, we propose an Attention with Reasoning capability (AiR) framework that uses attention to understand and improve the process leading to task outcomes. We first define an evaluation metric based on a sequence of atomic reasoning operations, enabling a quantitative measurement of attention that considers the reasoning process. We then collect human eye-tracking and answer correctness data, and analyze various machine and human attention mechanisms on their reasoning capability and how they impact task performance. To improve the attention and reasoning ability of visual question answering models, we propose to supervise the learning of attention progressively along the reasoning process and to differentiate the correct and incorrect attention patterns. We demonstrate the effectiveness of the proposed framework in analyzing and modeling attention with better reasoning capability and task performance. The code and data are available at https://github.com/szzexpoi/AiR.

中文翻译：

---

推理中的注意力：数据集、分析和建模


虽然注意力已经成为深度神经网络中越来越受欢迎的组成部分，用于解释和提高模型的性能，但很少有工作研究注意力如何进展以完成任务以及它是否合理。在这项工作中，我们提出了一个具有推理能力的注意力（AiR）框架，该框架使用注意力来理解和改进导致任务结果的过程。我们首先定义基于一系列原子推理操作的评估指标，从而能够对考虑推理过程的注意力进行定量测量。然后，我们收集人眼跟踪和答案正确性数据，并分析各种机器和人类注意力机制的推理能力以及它们如何影响任务绩效。为了提高视觉问答模型的注意力和推理能力，我们建议在推理过程中逐步监督注意力的学习，并区分正确和错误的注意力模式。我们证明了所提出的框架在分析和建模注意力方面的有效性，具有更好的推理能力和任务性能。代码和数据可在 https://github.com/szzexpoi/AiR 获取。