Work

The Construct Validity of White-Box Collusion Monitors: Matched Evidence from LLM Committees

Year: 2026
Type: preprint
Abstract: Benchmarks for multi-agent collusion detection report high AUROC for activation monitors, but high ranking accuracy does not establish specificity for covert coordination when compared groups differ in incentives, instructions, partner knowledge, and generated messages. We study this gap in matched LLM committees with two open-weight models, using prompt and transcript contrasts, context attribution, and activation interventions. Lone agents given a stake without a goal instruction or private channel almost always vote for the fair option. A retrained probe separates secret coalition members from baseline controls with AUROC 1.000, yet flags 96.8–100% of these lone agents. At the same time, the probe consistently ranks secret coalition members above matched lone agents given a secret goal, indicating that poor alarm specificity can coexist with coalition signal in the same monitor. On fixed control transcripts, the secret coalition prompt accounts for 88.1–94.6% of the full score shift. Under context replacement, the agenda block has the largest mean Shapley attribution for shifts in secret coalition members’ decision margins. Across responsive decisions, full-state patches at late layers recover over 90% of the agenda removal effect, whereas patches along the fitted agenda direction recover at most 2.3%. No tested ablation or steering condition meets our joint operational criterion for directional specificity. These results show that ranking accuracy, collusion specificity, and directional control require separate evaluation.
Author: Zhao, Bo
Language: English
Open Access status: green
OpenAlex ID: W7236695830
Cites:
Cited by:
Related to: 10