Beyond LLM-as-a-Judge: Independence-Aware Heterogeneous Evaluation with Conflict-Gated Verification
Year: 2026
Type: article
Abstract: Recent empirical studies reveal that large language model (LLM) evaluator ensembles exhibit substantial correlated errors, demonstrating that the independence assumptions underlying classical Condorcet aggregation do not hold for LLM panels. A diverse nine-judge frontier panel has been shown to provide information equivalent to approximately 2.2 independent votes. Furthermore, highly self-consistent frontier models often exhibit weak consistency-correctness correlation, and instruction-tuned judges collapse toward random-guessing on objective benchmarks like JudgeBench when stylistic confounds are controlled. Building on these findings, we propose a structural shift from monolithic semantic scoring to heterogeneous, multi-paradigm evidence aggregation. We formulate an independence-aware Evaluator Conflict Score (ECS) that dynamically weights epistemic disagreement across probabilistic semantic, deterministic symbolic, programmatic, and retrieval-based verification mechanisms. We present a conflict-gated cascading architecture that treats disagreement as a trigger for investigative escalation, ensuring that evaluation rigor scales proportionally to uncertainty.
Author: Kumar, Sourav
Institution: SRM Institute of Science and Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7208811800
FWCI:
Cites:
Cited by:
Related to: