Broken Input Looks Exactly Like Bad Reasoning
Year: 2026
Type: preprint
Abstract: When a language model pipeline produces a wrong judgment, the field's default diagnosis is that the model reasoned badly. This paper documents four cases from a production legal citation verification system where that diagnosis would have been wrong. In each, the model reasoned correctly and consistently over text it had never been shown in complete or accurate form. A text-handling defect upstream of the model produced the wrong output — a hyphen, a character cap, a sentence boundary, and a privacy filter. The distinction matters because the two failure modes require opposite fixes and are observationally identical from the output side. The fourth case is a distinct and more important sub-class: the corruption was not a defect at all. A privacy-redaction pass working exactly as designed destroyed the evidence it existed to make safe to analyse, and the damage propagated into a second subsystem where it silently disabled a human correction. All four are named against public opinions, with the specific text, so the claims can be checked.
Author: SCHULZ, BENJAMIN
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7204024556
Cites:
Cited by:
Related to: