A review of visualisation-as-explanation techniques for convolutional neural networks and their evaluation

Visualisation techniques are powerful tools to understand the behaviour of Artificial Intelligence (AI) systems. They can be used to identify important features contributing to the network decisions, investigate biases in datasets, and find weaknesses in the system's structure (e.g., network architectures). Lawmakers and regulators may not allow the use of smart systems if these systems cannot explain the logic underlying a decision or action taken. These systems are required to offer a high level of 'transparency' to be approved for deployment. Model transparency is vital for safety–critical applications such as autonomous navigation and operation systems (e.g., autonomous trains or cars), where prediction errors may have serious implications. Thus, being highly accurate without explaining the basis of their performance is not enough to satisfy regulatory requirements. The lack of system interpretability is a major obstacle to the wider adoption of AI in safety–critical applications. Explainable Artificial Intelligence (XAI) techniques applied to intelligent systems to justify their decisions offers a possible solution. In this review, we present state-of-the-art explanation techniques in detail. We focus our presentation and critical discussion on visualisation methods for the most adopted architecture in use, the Convolutional Neural Networks (CNNs), applied to the domain of image classification. Further, we discuss the evaluation techniques for different explanation methods, which shows that some of the most visually appealing methods are unreliable and can be considered a simple feature or edge detector. In contrast, robust methods can give insights into the model behaviour, which helps to enhance the model performance and boost the confidence in the model's predictions. Besides, the applications of XAI techniques show their importance in many fields such as medicine and industry. We hope that this review proves a valuable contribution for researchers in the field of XAI.