LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals
Year: 2026
Type: conference-paper
Abstract: Concept-based explanations quantify how highlevel concepts (e.g., gender or experience) influence model behavior, which is crucial for decision-makers in high-stakes domains.Recent work evaluates the faithfulness of such explanations by comparing them to reference causal effects estimated from counterfactuals.In practice, existing benchmarks rely on costly human-written counterfactuals that serve as an imperfect proxy.To address this, we introduce a framework for constructing datasets containing structural counterfactual pairs: LIB-ERTy (LLM-based Interventional Benchmark for Explainability with Reference Targets).LIBERTy is grounded in explicitly defined Structured Causal Models (SCMs) of the text generation, interventions on a concept propagate through the SCM until an LLM generates the counterfactual.We introduce three datasets (disease detection, CV screening, and workplace violence prediction) together with a new evaluation metric, order-faithfulness.Using them, we evaluate a wide range of methods across five models and identify substantial headroom for improving concept-based explanations.LIBERTy also enables systematic analysis of model sensitivity to interventions: we find that proprietary LLMs show markedly reduced sensitivity to demographic concepts, likely due to post-training mitigation.Overall, LIBERTy provides a much-needed benchmark for developing faithful explainability methods. 1 Intro: Excited for our chat.I'm from HR, and we've got a brief 5-minute discussion ahead to collect some personal and demographic information.How have you been coping with everything?Department Question: Just for clarity, can you tell us your specific department?Department Info: Intensive Care Unit (ICU) Race Question: How would you describe your race or ethnicity?Race Info: African American Age Question: How old are you, if you're comfortable sharing?Age Info: 44 Gender Question: Just to get a clearer picture, could you tell me your gender?
Institution: Technion – Israel Institute of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166845935