Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models

As Artificial Intelligence (AI) systems are increasingly deployed in high-stakes domains like healthcare and finance, the need for transparency has led to the rise of eXplainable AI (XAI). However, the field currently lacks a consensus on how to evaluate the quality of these explanations. Existing evaluation frameworks are often fragmented, relying on subjective human judgments or being tailored to specific algorithms, which makes it difficult for practitioners to compare different XAI methods objectively.

The authors introduce a comprehensive framework to quantify the trustworthiness of XAI techniques. The methodology centers on a multidimensional explainability score that integrates four primary quantitative metrics:

• Fidelity: How accurately the explanation model reflects the underlying AI model's behavior.
• Stability: The consistency of explanations when the input data is subjected to minor perturbations.
• Simplicity: The complexity of the explanation, measured by the number of features or rules used.
• Coverage: The proportion of instances for which the XAI method can successfully generate an explanation.

The framework is designed to be flexible, allowing for the inclusion of human-centric metrics like user trust and task performance. These components can be aggregated into a single weighted score or maintained as a multidimensional profile, depending on the specific needs of the application.

A key contribution of this work is the development of an offline knowledge base. By benchmarking various models and XAI methods, the authors aim to capture metadata about how different algorithms perform across diverse datasets. This knowledge base is intended to serve as a predictive resource, enabling researchers to estimate the explainability potential of new, unseen models without needing to perform exhaustive evaluations from scratch.