Explainability is one of the key elements for building trust in AI systems. Among numerous attempts to make AI explainable, quantifying the effect of explanations remains a challenge in conducting human-AI collaborative tasks. Aside from the ability to predict the overall behavior of AI, in many applications, users need to understand an AI agent's competency in different aspects of the task domain. In this paper, we evaluate the impact of explanations on the user's mental model of AI agent competency within the task of visual question answering (VQA). We quantify users' understanding of competency, based on the correlation between the actual system performance and user rankings. We introduce an explainable VQA system that uses spatial and object features and is powered by the BERT language model. Each group of users sees only one kind of explanation to rank the competencies of the VQA model. The proposed model is evaluated through between-subject experiments to probe explanations' impact on the user's perception of competency. The comparison between two VQA models shows BERT based explanations and the use of object features improve the user's prediction of the model's competencies.

中文翻译：

---

VQA中解释对AI能力预测的影响

可解释性是在 AI 系统中建立信任的关键要素之一。在使 AI 可解释的众多尝试中，量化解释的效果仍然是执行人与 AI 协作任务的挑战。除了能够预测 AI 的整体行为之外，在许多应用中，用户还需要了解 AI 代理在任务域不同方面的能力。在本文中，我们评估了解释对用户在视觉问答 (VQA) 任务中 AI 代理能力的心理模型的影响。我们根据实际系统性能和用户排名之间的相关性，量化用户对能力的理解。我们引入了一个可解释的 VQA 系统，该系统使用空间和对象特征，并由 BERT 语言模型提供支持。每组用户只看到一种解释来对 VQA 模型的能力进行排名。所提出的模型通过主体间实验进行评估，以探讨解释对用户能力感知的影响。两个 VQA 模型之间的比较显示了基于 BERT 的解释，并且对象特征的使用提高了用户对模型能力的预测。