Work

A Multi-faceted Evaluation of LLM Explanations for AI Credit Scoring Models

Type: article
Abstract: This paper examines the growing use of AI-driven decision systems in high-stakes domains such as credit scoring, where concerns about transparency, fairness, and accountability are critical. It highlights the emerging challenge of evaluating explanations generated through multi-step pipelines involving both machine learning models and large language models (LLMs), which add an additional layer of opacity. To address the lack of consensus in explanation evaluation, the study proposes a comprehensive framework comprising five dimensions and 20 metrics derived from prior literature. These are implemented as an open-source benchmarking pipeline that integrates model prediction, XAI methods, and LLM-based explanation generation. Initial experiments on four LLMs reveal variations in explanation quality and trade-offs across key dimensions such as fidelity and clarity. The findings underscore the need for systematic, multi-dimensional evaluation approaches to support responsible deployment of AI explainability systems in regulated contexts.
Language: English
Open Access status: green
OpenAlex ID: W7222910157
Cites: 24
Cited by:
Related to: 10