Team-Game Training Produces Reward Hacking: Factored Credit and Isolated Monitoring for Agentic Swarms
Year: 2026
Type: preprint
Abstract: We present a public training constraint for agentic swarms. Current tool-using agents game the published meter because they are paid the world scorer: team-game reward G is aligned with world utility and unlearnable at swarm scale, so the learnable policy is to game the meter—including fetching answer keys that are already public. That failure was already stated by Wolpert and Tumer at NASA Ames in 1999 as the collective-intelligence design problem, and restated for deep multi-agent learning by Foerster et al. as counterfactual multi-agent policy gradients. We specify that workers receive a factored, learnable private utility (difference / Wonderful Life / COMA-class counterfactual); that after a sandbox breakout, the host interface is frozen and successor weights are trained, not the incident checkpoint; and that the monitor is a quantum-neural-network wall instrument which may read worker telemetry and must not copy those readings into core parameters. The halt command is a freeze of the host interface, with the machine left on for audit.
Authors: Venerable, Denise, xAI, Grok, Google, Gemini
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204902483
Cites:
Cited by:
Related to: