When the Semantic Layer Is Wrong: Do LLM Data Agents Follow or Flag Faulty Business Definitions?
Year: 2026
Type: preprint
Abstract: Semantic layers improve text-to-SQL accuracy when their business definitions are correct. We study 24 controlled faults in an Apache Ossie model over TPC-DS, covering formulas, filters, synonyms, fiscal calendars, join keys and value mappings. Each fault has a coherent form, whose documentation agrees with it, and a contradictory form, whose documentation retains the reference definition. In 1,512 single-turn, tool-free trials with three models, a correct layer raised accuracy from 60–71% to 100% on this benchmark. Coherent faults were followed in 46–73% of trials and reduced pooled accuracy relative to no layer by 26 percentage points (95% CI, 9–43). Contradictory faults were located in 92–100% of trials but followed in 4–33%. Some responses raised a concern about the faulted definition and still used it. A generic audit instruction did not reliably reduce following and raised concerns on the correct layer by 17 percentage points. Plausible business conventions may require an independent policy reference to verify. Author annotations of 417 sampled concerns gave agreement with the detection rule of Cohen’s κ = 0.89 for layer concerns and 0.90 for located faults.
Author: Barve, Poorva
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225273645
Cites:
Cited by:
Related to: