Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective
Year: 2026
Type: conference-paper
Abstract: Compositional generalization tests are often used to estimate the compositionality of LLMs.However, such tests have the following limitations: (1) they only focus on the output results without considering LLMs' understanding of sample compositionality, resulting in explainability defects; (2) they rely on dataset partition to form the test set with combinations unseen in the training set, suffering from combination leakage issues.In this work, we propose a novel rule-generation perspective for compositionality estimation for LLMs.It requires LLMs to generate a program as rules for dataset mapping and provides estimates of the compositionality of LLMs using complexity-based theory.The perspective addresses the limitations of compositional generalization tests and provides a new way to analyze the compositionality characterization of LLMs.We conduct experiments and analysis of existing advanced LLMs based on this perspective on a stringto-grid task, and find various compositionality characterizations and compositionality deficiencies exhibited by LLMs.Our ode is available at https://github.com
Authors: Ziyao Xu, Cong Wang, Houfeng Wang
Institution: Peking University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166883690