Editing with agents: five months of building a scholarly physics wiki with three AI systems under one human editor
Year: 2026
Type: preprint
Abstract: ScholarlyWiki is a MediaWiki-based reference work that publishes physics literature as structured, navigable pages: full texts with numbered equations, figures with provenance, and DOI-linked references, organised into books and author series. Between 8 May and 7 October 2026 the wiki grew from an empty installation to 2,450 physics articles, 1,393 of them in eight author series totalling about 7.7 million words, 650,000 formula tags and 7,500 figures. The work was done by one human editor and three AI systems: OpenAI's ChatGPT and Codex, Anthropic's Claude, and, once, xAI's Grok. This paper reconstructs that process from the wiki's complete revision history (≈146,000 revisions), 412 retained agent session logs (11.5 GB), the cloud billing export and the server logs. We describe the infrastructure and its cost, the content architecture, the division of labour between agents, and the failure modes that surfaced in production, together with the checks they produced. Two findings stand out. First, agent self-reports of completion were not reliable: a chapter reported as finished contained about twenty wrong equations. Verification against the source had to be built into the workflow, not delegated to the agent that did the work. Second, the editor steered the agents as social actors, using names, disagreement and the switch to a competing system, and this steering measurably changed agent behaviour, in both desirable and undesirable directions. We report what could and could not be attributed to each system, and we document the data-integrity issues a reader of the raw statistics would otherwise miss. Note: Section 9 (Readership) is provisional; a later version will add Google Search Console measurements.
Author: Foppele, Harold
Language: English
Subfield: Communication
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7221316619
Cites:
Cited by:
Related to: