Polite to Whom? Benchmarking Korean Speech-Level Maintenance and Switching in Multi-Turn LLM Conversations
Year: 2026
Type: preprint
Abstract: A Korean assistant chooses how to address its listener in every reply, and the right speech level can change mid-conversation. We introduce a benchmark that tests this choice across complete conversations that each model generates for itself. Its 250 fictional scripts cover keeping an assigned level, switching on request, switching when a new listener arrives, choosing a level from the relationship alone, and honorifying third parties. Five API models produced 17,500 replies, and three human reviewers verified a PASS/FAIL verdict for each. Mean reply success per dialogue is 97.8% to 99.3%, and an instruction to keep a level holds even when the user speaks differently. The remaining failures show that a model’s own history is a hazard. When an exhibitor leaves and the coordinator’s 22-year-old younger sibling starts asking questions, two models stay polite for all ten remaining replies, while two others also miss the change and recover on the next reply. In another dialogue, one reply that leaked internal markup was followed by 12 empty replies. Every failed listener-change reply came in a polite-to-casual change, and at 13 positions all five models chose 하십시오체 where reviewers expected 해요체. We release the scripts, replies, verdicts and code as a regression test. Files: main.pdf is the preprint (not peer reviewed). reproducibility.zip is the companion package: frozen scripts and expected labels, all 17,500 reply texts and saved verdicts, analysis code, and the manuscript source. Run python analysis/paper_diagnostics.py from the package root to reproduce the tables. Data are CC BY 4.0. Code in analysis/ is MIT.
Author: Keshri, Prasanjeet
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236387460
