Work

Confidence as a Tie-Breaker: Reassessing Multilingual Hedging Bias in LLM-as-a-Judge Evaluation

Year: 2026
Type: conference-paper
Abstract: LLM judges are often used to score generated answers, but their decisions may be affected by surface style rather than semantic correctness.We introduce PolyJudge-Uncertain, a controlled benchmark for studying multilingual hedging effects in LLM-as-a-judge evaluation.The benchmark contains 5,120 short factual QA instances across English, Hindi, Hinglish, and Bengali, balancing assertive versus hedged style and correct versus incorrect answers.A small pilot suggested a large pointwise penalty against hedged answers.After repairing multilingual templates and adding quality-control checks, this pointwise effect largely disappears: final pointwise accuracy is 99.8%, with no meaningful assertive-hedged gap.The robust remaining effect is pairwise: when two answers are equally correct and differ only in style, the judge prefers the assertive answer in 1,276 of 1,280 cases.We interpret this as a protocol-and task-specific assertiveness preference, not as a universal bias against hedging.Our findings highlight benchmark auditing as a central requirement for multilingual judge-bias research.
Language: English
Open Access status: hybrid
OpenAlex ID: W7166807404
FWCI:
Cites: 31
Cited by:
Related to: 10