Can AI reason from inside a tradition? A preregistered benchmark for tradition fidelity in language models
Year: 2026
Type: article
Abstract: Language models are increasingly asked to speak for cultures, religions and other ways of knowing, and recent work suggests they often adopt a tradition's voice while reasoning from a narrower default. A 2026 preprint describes a "persona illusion" on ethical dilemmas rooted in Indigenous contexts; Agarwal and colleagues find that an average American is a closer match to Indian cultural values than any Indian-trained model they tested, and that prompting does not meaningfully improve alignment. Whether this gap can be closed by architecture, and not only by instructions, is untested. We propose the WhatIfWe Open Benchmark, which compares three conditions on problems where traditions are known to reason differently: tradition-grounded agents, whose retrieval is partitioned by tradition and whose claims must cite their sources; a single model prompted to answer as each tradition in turn; and a single model answering for all traditions at once. Practitioners formed in each tradition judge the outputs blind for identification and for fidelity. The benchmark is preregistered, needs only published texts used with permission, and carries a stop condition: if grounded agents are not identified more often than the prompted single model, the hypothesis is falsified and the result is published. Working paper, not peer reviewed; comments welcome.
Author: Ibarra, Hector
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225318809