A survey of visual analytics for Explainable Artificial Intelligence methods

Highlights

• A comprehensive survey for visual analytics, particularly those that adopted explainable artificial intelligence (XAI) methods, in interpreting Neural Networks is conducted.

• We reviewed the literature based on model usage and visual approaches.

• We concluded some visual approaches commonly used to support the illustration of XAI methods for various types of data and machine learning models; however, a generic approach is needed for the field.

• We listed several future work including data manipulations, scalability, and bias in data representation, and generalizable real-time visualizations integrating XAI.

Abstract

Deep learning (DL) models have achieved impressive performance in various domains such as medicine, finance, and autonomous vehicle systems with advances in computing power and technologies. However, due to the black-box structure of DL models, the decisions of these learning models often need to be explained to end-users. Explainable Artificial Intelligence (XAI) provides explanations of black-box models to reveal the behavior and underlying decision-making mechanisms of the models through tools, techniques, and algorithms. Visualization techniques help to present model and prediction explanations in a more understandable, explainable, and interpretable way. This survey paper aims to review current trends and challenges of visual analytics in interpreting DL models by adopting XAI methods and present future research directions in this area. We reviewed literature based on two different aspects, model usage and visual approaches. We addressed several research questions based on our findings and then discussed missing points, research gaps, and potential future research directions. This survey provides guidelines to develop a better interpretation of neural networks through XAI methods in the field of visual analytics.

Introduction

Machine learning (ML) techniques have achieved impressive performance in various domains such as medicine, finance, and autonomous vehicle systems with advances in computing power and technologies [1], [2]. Neural Network (NN), as a sub-branch of ML, has become a powerful technique in finding complex patterns in high-dimensional datasets and providing high prediction accuracy in many domains [3]. However, NN-based models have a complex structure, which makes it difficult for them to be interpreted and understood. NNs are considered black-box models since their inner working and decision-making mechanisms are not understandable by a human. This reveals one of the most important issues in black-box models: transparency and explainability [4].

End-users often want to understand how a classifier makes predictions, particularly in sensitive domains, such as healthcare, transportation, defense, and finance, where decision making often has a critical impact. Explaining how predictions are made by ML models by clarifying their working mechanisms would increase trustworthy of ML models. To address this important need, interpretable ML algorithms has been developed rapidly in understanding the inner working mechanisms of black-box models [5]. One of the most important efforts is the development of a re-emerging field in eXplainable Artificial Intelligence (XAI) [6]. According to Defense Advanced Research Projects Agency (DARPA) technical report [6], XAI is defined as “a suite of machine learning techniques that enables human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners”. Although interpretability and explainability are often used interchangeably by the ML community, there are slight differences in the definition of interpretable ML and explainable AI. Miller [7] defines interpretability as “the degree to which an observer can understand the cause of a decision” and equates interpretability and explainability definitions. From the ML context, interpretability can be defined as understanding how the decision/prediction is given by machine learning algorithms with reasoning. The term explainability is more related to the internal working mechanisms of black-box models. Therefore, XAI reveals the internal functioning of black-box models and the rationale behind the decisions through various methods. While domain experts who are inexperienced in ML often want to understand through reasoning and cause–effect relationship why a certain decision has been made, ML scientists focus on the internal working mechanisms of ML models and try to understand how their components contribute to certain predictions. XAI aims to help end-users and domain experts to gain insight into how black-box models make predictions. It also helps ML scientists with the model development process by explaining the decision-making process of the black-box models.

Visual analytics (VA) is inherent way to represent the data/ model understandably, particularly to those who are inexperienced in ML. VA has been often used in providing interpretable ML models by understanding [1], [5], diagnosing [8], [9], and steering [10] the model and underlying data through an interactive visual interface. Combining the techniques of VA with XAI algorithms would present an ideal platform to clarify the black box structure of ML. However, there are only a few recent works combining VA with the current stage of XAI methods to provide explainable ML models to humans. So, we target the audience of this review to following groups: (1) VA scientists who would like to adopt XAI methods to interpret NN, (2) ML scientists, particularly in the field of XAI, who may need VA to interpret their work, and (3) end-users/domain experts who use NN for data classification and predications. This survey paper aims to find current trends and challenges of VA in interpreting black-box models by adopting XAI methods and present future research directions in this area. Within the study, we would like to discover and present how VA can support a better interpretation of NN models with XAI methods.

The explainability and interpretability of black-box models are recent hot topics, and many studies have been done in the field. Most of the studies have been focused on the interpretation of NNs due to their state-of-the-art performance in various domains. Therefore, we limit our study by reviewing papers that focus on the interpretation of NNs among black-box models. There are several literature reviews of XAI methods [4], [11], [12], [13], [14], and visual analytics on interpretable machine learning [2], [15], [16], [17], [18], [19], [20] respectively. However, to our knowledge, there is no literature review that focus on VA research combined with XAI methods. Such review will help to analyze potential future research directions to develop a better interpretation of neural networks through XAI methods in the field of visual analytics. Therefore, we reviewed 55 papers that contributed to the interpretation of NN models via visual analytics with and without XAI methods in terms of model usage and visual approach. Model usage refers to techniques that are used to explain NN models in the fields of VA and XAI respectively. The visual approach mainly focuses on analyzing how visualization techniques are used in data and architecture representations, performance analysis, and local and global explanations. Our main contributions are as follows:

• We present a review of VA research in interpreting deep learning with a focus on with and without adopting XAI methods.

• We reviewed the literature based on (1) model usage in visual interpretation and XAI algorithms respectively, and (2) visual approach where commonly used visual approaches are summarized.

• We highlight the current trends and limitations, and discuss future research directions of VA that adopts XAI for NN models.

The rest of the paper is organized as follows: Section 2 provides theoretical background about black-box models and XAI methods. Section 3 shows the methodology of this review. Section 4 reviews visual interpretation papers and Section 5 reviews visual-based XAI papers based on the model usage and visual approaches, respectively. Section 6 states the current trends and discusses future directions of VA for XAI. Section 7 concludes the paper.