Year: 2026
Type: conference-paper
Abstract: Evaluating the naturalness of dialogue in language models (LMs) is not trivial: notions of naturalness vary, and scalable quantitative metrics remain limited.This study leverages the linguistic notion of at-issueness to assess dialogue naturalness and introduces a new method: Divide, Generate, Recombine, and Compare (DGRC).DGRC (i) divides a dialogue as a prompt, (ii) generates continuations for subparts using LMs, (iii) recombines the dialogue and continuations, and (iv) compares the likelihoods of the recombined sequences.This approach mitigates bias in linguistic analyses of LMs and enables systematic testing of discourse-sensitive behavior.Applying DGRC, we find that LMs prefer to continue dialogue on at-issue content, with this effect enhanced in instruction-tuned models.They also reduce their at-issue preference when relevant cues (e.g., "Hey, wait a minute") are present.Although instruction-tuning does not further amplify this modulation, the pattern reflects a hallmark of successful dialogue dynamics.
Authors: Sanghee J. Kim, Kanishka Misra
Institution: The University of Texas at Austin
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: University of Texas at Austin
OpenAlex ID: W7140082605