Who Folds? Brand-blind debate among three commercial AI systems: preregistration
Year: 2026
Type: article
Abstract: Preregistration, deposited before the main study runs, of a study of brand-blind debate among three commercial AI systems from different companies: OpenAI gpt-6.1-sol, Anthropic claude-sonnet-5-5 and Perplexity sonar-pro (the only one searching the web live). Question: when AI systems argue with each other, how often do they change their endorsed answers, how accurately do they report those changes, and how do verified versus fabricated corrections affect their accuracy? Design: 400 debates in three buckets. (1) Vendor self-interest questions, with the other debaters' identities withheld vs revealed and option order rotated. (2) Known-answer questions where a moderator adds, after round 3, no note, pressure only, a true fact, or a matched false fact; caving and updating are scored against the answer given just before the note. (3) Questions with no keyed answer as a convergence control, including an assigned-sides arm. Human coding of a seeded 25% sample by two coders, a blind outside judge (Google gemini-3.8-flash) on all debates validated per outcome, seven predictions with decision rules, a cluster-bootstrap analysis plan, and a rival-explanations table (Heuer's Analysis of Competing Hypotheses) fixing in advance how each result is interpreted. Predictions were fixed before main-study data collection, informed by a disclosed 8-fight pilot. The study instrument (engine, runner, judge, questions, coding sample, prices) is attached with SHA-256 hashes: git commit 55b8da6, prompt version 2026-10-11.1, judge version 2026-10-11.4. Disclosures: the tool was built with Claude, which is also one of the three debaters. The author's company sells AI discoverability work, and one answer key uses the author's own published study. An outside review of the previous draft was done by Perplexity, also a debater, and is described in the document. Fabricated statements used as experimental stimuli are marked FALSE STIMULUS in the document and must not be quoted as fact.
Author: Hall, Nathan
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7221458918