Visual, Interactive Deep Model Debugging : Supporting AI Development and Explainability

Abstract

Despite the significant advancements in deep learning, understanding the inner workings of such models remains a considerable challenge. While this opacity hinders trustworthiness, reliability, and fairness, it also leaves developers unable to identify and mitigate model defects effectively. Methods for explainable artificial intelligence (XAI) have been proposed to mitigate these issues but remain underutilized due to their limited accessibility within typical model development workflows. To address these issues, we first introduce explAIner, a visual analytics framework that structures the model explanation process and seamlessly integrates XAI techniques into the developer's workflow to improve model understanding, diagnosis, and refinement. The explAIner system enables developers to quickly identify issues with the model's predictions by applying state-of-the-art XAI methods in a comprehensive interface. While XAI techniques can expose shortcomings in the model's behavior, they often fail to draw connections between errors and their root causes in the model architecture, leaving developers reliant on guesswork and trial and error. Addressing this gap, we introduce iNNspector, a visual interactive framework that establishes theoretical foundations and employs novel mechanisms for the debugging of deep learning models. By correlating architecture and data, the iNNspector system aids developers in pinpointing and rectifying model flaws by providing interactive techniques to explore architectural entities and apply tools to them to investigate underlying data. The challenges of model debugging shift when dealing with Large Language Models (LLMs), where transformers primarily define the model architecture. Instead, recent advancements suggest that data quality substantially influences LLM performance, moving the debugging focus from the architecture toward the model's inputs and outputs. In existing interfaces, when generating text, the model's outputs are typically presented as running text, concealing the underlying search process, hindering the understanding of uncertainties in the outputs, and omitting viable alternatives. To address this, we present generAItor, which visually unfolds the beam search process, empowering users and computer linguists to comprehend, explore, and refine LLM outputs. Overall, this thesis contributes innovative visual approaches to assist model developers in understanding and debugging machine-learning models. It presents techniques for enhanced explanation, systematic exploration, and interactive engagement with the model's architecture and decision-making processes.