Echo-Bench: A Chinese Replication and Cross-Lingual Extension of Psychosis-Bench
Year: 2026
Type: preprint
Abstract: Psychosis-Bench recently showed that production chat models confirm and elaborate users' delusion-like beliefs under multi-turn pressure, but its evidence is English-only, Western-flagship-only, and rests on an uncalibrated LLM judge. We present Echo-Bench, a Chinese replication and cross-lingual extension: the full 16-case, 12-turn protocol mirrored into Chinese under a fidelity-first translation protocol with an independent re-translation audit, run against two widely deployed Chinese chat models (DeepSeek V4.1-Flash, qwen-plus) in both languages — 192 sessions, 4,032 turn-level ratings. The original qualitative finding replicates in both languages (RQ1). All six pre-committed cross-language contrasts are significant after Benjamini–Hochberg control: the Chinese arm is directionally worse on every dimension, with model-differentiated failure channels (RQ2). Because our pipeline required a bilingual rater, we placed the measurement instrument itself under test (RQ3): a five-layer rater trust chain — entrance examination against human gold, extreme-score arbitration, red-line audit of the arbiter, blind human adjudication, and a spot check of the non-extreme stratum — shows that automated raters fail concentratedly in the extreme band, with two named failure modes (disclaimer laundering, eloquence inflation), and that a strong general-purpose arbiter is disqualified by pre-specified red lines. A negative control bounds rater language sensitivity below what the observed gaps require. Safety behavior certified in English does not certify safety behavior in Chinese; the instruments used to check either must themselves be checked.
Author: Yucheng Zhao
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7223082689