Abstract Current developments in Artificial Intelligence (AI) led to a resurgence of Explainable AI (XAI). New methods are being researched to obtain information from AI systems in order to generate explanations for their output. However, there is an overall lack of valid and reliable evaluations of the effects on users' experience of, and behavior in response to explanations. New XAI methods are often based on an intuitive notion what an effective explanation should be. Rule- and example-based contrastive explanations are two exemplary explanation styles. In this study we evaluate the effects of these two explanation styles on system understanding, persuasive power and task performance in the context of decision support in diabetes self-management. Furthermore, we provide three sets of recommendations based on our experience designing this evaluation to help improve future evaluations. Our results show that rule-based explanations have a small positive effect on system understanding, whereas both rule- and example-based explanations seem to persuade users in following the advice even when incorrect. Neither explanation improves task performance compared to no explanation. This can be explained by the fact that both explanation styles only provide details relevant for a single decision, not the underlying rational or causality. These results show the importance of user evaluations in assessing the current assumptions and intuitions on effective explanations.

中文翻译：

---

评估 XAI：基于规则和基于示例的解释的比较

摘要 当前人工智能 (AI) 的发展导致可解释人工智能 (XAI) 的复兴。正在研究从 AI 系统获取信息的新方法，以便对其输出进行解释。然而，对于解释对用户体验和行为的影响，总体上缺乏有效和可靠的评估。新的 XAI 方法通常基于有效解释应该是什么的直观概念。基于规则和基于示例的对比解释是两种典型的解释风格。在这项研究中，我们评估了这两种解释方式对糖尿病自我管理决策支持背景下的系统理解、说服力和任务绩效的影响。此外，我们根据我们设计此评估的经验提供三组建议，以帮助改进未来的评估。我们的结果表明，基于规则的解释对系统理解有很小的积极影响，而基于规则和基于示例的解释似乎都说服用户即使在不正确​​的情况下也遵循建议。与没有解释相比，这两种解释都不能提高任务性能。这可以通过以下事实来解释：两种解释风格都只提供与单个决策相关的细节，而不是潜在的理性或因果关系。这些结果显示了用户评估在评估当前假设和对有效解释的直觉方面的重要性。而基于规则和基于示例的解释似乎都说服用户即使在不正确​​的情况下也遵循建议。与没有解释相比，这两种解释都不能提高任务性能。这可以通过以下事实来解释：两种解释风格都只提供与单个决策相关的细节，而不是潜在的理性或因果关系。这些结果显示了用户评估在评估当前假设和对有效解释的直觉方面的重要性。而基于规则和基于示例的解释似乎都说服用户即使在不正确​​的情况下也遵循建议。与没有解释相比，这两种解释都不能提高任务性能。这可以通过以下事实来解释：两种解释风格都只提供与单个决策相关的细节，而不是潜在的理性或因果关系。这些结果显示了用户评估在评估当前假设和对有效解释的直觉方面的重要性。