LOCUS: Measuring Diagnostic Commitments, Actions, and Abstention
Year: 2026
Type: preprint
Abstract: A diagnostic benchmark can credit a correct label even when the chosen experiment supplies no evidence for that label. LOCUS makes this distinction testable using paired mechanism failures with identical passive observations and a unique distinguishing intervention. It separately scores submitted candidate mechanisms, exact predictions, actions and post-reveal localization. In a historical panel of three proprietary variants, action accuracy is 192/360 (53.33%) and full pre-intervention commitment is 3/360 (0.83%). Because full commitment includes action correctness, their gap alone cannot diagnose a conceptual deficit. More directly, of 236 hidden-label matches, 88 arise without distinguishing evidence. A separate Gemini native-tool contrast yields strict commitment success of 0/120 without tools and 9/120 with tools, with a registered intermediate or indeterminate interpretation. Exploratory removal of one enclosing Markdown fence changes these counts to 4/120 and 62/120; 52 tool-arm jobs still have no final text. These results motivate reporting evidential adequacy and output acceptance alongside accuracy. They characterize submitted behavior in a controlled synthetic task, not an intrinsic causal-reasoning limitation or transfer to real-world diagnosis. This deposit contains an author-named working preprint and a compact manuscript-source supplement with the tables, figures, and synthetic worked example needed to rebuild and inspect the paper. The supplement is not a complete response-level reproduction archive: it excludes private model-response corpora, held-out reserve data, credentials, human-participant data, and internal research records. Confirmatory results and post-hoc sensitivity analyses remain distinguished in the manuscript. No peer-review acceptance is claimed.
Authors: Noori, Ali, Bius, Mohammad
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222903276
Cites:
Cited by:
Related to: