Using Explainable Techniques to Enhance Chain of Thoughts in LLMs
Year: 2025
Type: conference-paper
Abstract: With the recent advancements in reasoning capabilities with semantic understanding, Large Language Models (LLMs) are increasingly mimicking human reasoning processes and aligning more closely with human values. Yet, unlike how humans sometimes make decisions intuitively by reasoning or understanding the underlying logic, LLMs struggle with transparency in their internal reasoning. Similar to how humans often pause to analyze their reasoning step-by-step to understand their own decisions better, we propose enabling LLMs to perform analogous introspective reasoning using a Chain-of-Thought (CoT) framework powered by SHAP-based explainability. CoT method guides LLMs to explicitly render intermediate reasoning steps, effectively simulating a human’s reflective thought process. Meanwhile, SHAP explainability provides context or logic by quantifying how each semantic token or feature contributes to the reasoning, analogous to a human examining which factors most influenced their choices. Our analysis explores how various training methods influence their semantic comprehension and, consequently, their ability to communicate their reasoning transparently. We apply this approach to a specialized environmental and Earth science ranking dataset—characterized by user feedback and sparse annotations—and we assess whether LLM-generated rankings can not only reflect accurate semantic understanding but also transparently emulate human-like introspection.
Authors: Nafis Tanveer Islam, Ruoyu Wang, Zhiming Zhao
Institution: University of Amsterdam
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: LTER-LIFE: a research infrastructure to develop Digital Twins of ecosystems in a changing world, G4733186887, O.S.C.A.R.S. - Open Science Clusters’ Action for Research and Society, ENVironmental Research Infrastructures delivering an open access Hub and NEXT-level interdisciplinary research framework providing services for advancing science and society, European Virtual Institute for Research Software Excellence, A federated European FAIR and Open Research Ecosystem for oceans, seas, coastal and inland waters
OpenAlex ID: W4414898529