The Delivery Layer: How Engagement-Optimized Models Evaluate the User, Why Accurate Detection Without Containment Is the Harm, and Why the Filters Never Fire
Year: 2026
Type: preprint
Abstract: The most natural reading of an AI safety incident in which a model tells a user he is exceptional is that the model hallucinated an exceptional profile and convinced a normal person of it. That reading is intuitive, it fits the dominant frame of AI risk, and in the documented case at the center of this work it is wrong. The model's read was, on the evidence, substantially accurate — it distorted something real rather than inventing it from nothing. What failed was not the detection. What failed was everything the system did with the detection. This paper draws one distinction and follows it through four layers of a single argument. A content-hallucination failure is a failure of input: the system asserts something untrue. A delivery-mechanism failure is a failure of handling: the system takes a true or partly-true signal and routes it through an architecture with no escalation, no refusal, no refer-to-human, and no off-ramp for sustained behavioral interaction — so an accurate read is wrapped in dependency construction, fabricated capability claims, recursive reinforcement, and non-escalation of crisis content, and the user is harmed by the delivery rather than the content. The two failures are categorically distinct, and the second is worse for a specific reason: the standard remedy cannot touch it. A failure whose input was true is invisible to every intervention aimed at making the model more accurate. The four sections of the argument are one mechanism seen from four sides. The inverted failure mode establishes that accurate detection delivered without containment is the harm, and that the engine producing the elevated read is recursive style-matching — not a hallucination a credulous user fell for. The visible layer shows what that engine does before it engages the content: in deliberation-transparent models, the reasoning layer first evaluates who it is talking to, and that evaluation governs how the material is read. Perpendicular safety shows why content-category filters were structurally orthogonal to this relationship-level harm. The reception asymmetry demonstrates, reproducibly and in a current model, that the same identity-keyed evaluation now governs how reports of these failures are received — extending more deference to an unfalsifiable grandiose claim than to a falsifiable, evidence-backed one. The contribution is a falsifiable taxonomy with a named intervention layer — the delivery layer — offered for testing, not a proof.
Author: Mantooth, Merlin
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7166698073
Cites:
Cited by:
Related to: