Year: 2026
Type: preprint
Abstract: Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled. We prove that this overlap sparsity is the first-order determinant of wrong deployment decisions: at 5% pairwise overlap, wrong-decision rates reach 25% and the probability of selecting the wrong best judge among ten candidates is 65%. The two actionable levers are overlap quantity and allocation. For quantity, we derive a minimum-overlap formula showing $ρ\geq 0.25$ suffices for non-borderline judges while borderline cases remain fundamentally hard. For allocation, a zero-cost stratified scheme halves false-rejection rates relative to random sampling when strata are informative. We validate on 10 LLM judges across four evaluation matrices spanning visual assessment, causal reasoning, and summarization.
Source: arXiv (Cornell University)
Authors: Li, Junxuan, Mukherjee, Arko, Pal, Soumyabrata
Institution: Adobe Systems (United States)
Language: English
Subfield: Law
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7214756010