Work

Linkage Is Not Validity: Auditing the Actual Basis of a Belief

Year: 2026
Type: preprint
Abstract: Linking a reason report to the right episode does not establish that it measures the reason actually used. Asking for a reason can prompt reconsideration, so a correctly linked report may describe a different inference. This paper develops a method for auditing a past basis. It models the probe separately from the original inference and shows when linked fallible indicators can be calibrated without a gold standard. Worked examples show how residual dependence and calibration drift undermine point identification and what can still be learned from bounded error or from the recorded overlap alone. A finite design criterion tells auditors which records must be available together for a given score; even all pairwise records can miss a three-way defect. An application to language-model faithfulness distinguishes an answer switch from actual reliance on a hint. The pump examples use synthetic data, and the simulator study is a proposed design rather than an empirical result.
Language: English
Open Access status: green
OpenAlex ID: W7233608605
Cites:
Cited by:
Related to: