Entity Exchange in the Wild: A Diagnostic Study of LLM Based Real-World Conversational Entity Extraction
Year: 2026
Type: conference-paper
Abstract: Entity extraction from spoken customer-agent conversations is increasingly driving automation in contact centers.Errors in extraction can trigger incorrect system actions such as database updates, verification failures, or workflow execution.While prior work has examined transcription noise and cross-turn reasoning, it has not systematically analyzed how entityexchange phenomena shape extraction performance.We model these phenomena along three orthogonal axes: Initiation (how an entity becomes relevant), Evolution (how commitment to the value of an entity develops across turns), and Articulation (how the final value of an entity is expressed in surface form).We evaluate 16 large language models on 6,387 realworld conversations spanning 12 entity types across numeric, alphanumeric, temporal, and free-text categories.Performance varies by up to 50-60% within the same LLM depending on the entity-exchange phenomena.The most severe failures occur when the entity value is revised during the interaction and the model must contrast intermediate values with a committed value.Even when no revision occurs, digit-bydigit and encoded expressions remain a consistent source of error.Error-Aware prompting improves extraction across all three axes, yielding average gains of upto 6.4% across all LLMs.Together, this work provides a structured basis for benchmarking entity extraction in real-world deployments and isolating systematic failure modes. Initiation Taxonomy Prompt INITIATION TAXONOMYModels how an entity first enters the conversation.I0.Elicited Entry Entity first enters the conversation in response to an explicit request/prompt.• Example: "May I have your order number?"→ "o nine five four"• Example: "Do you want your case number?"→ "three five seven six" I1.Volunteered Entry Entity is introduced proactively without a prior request/prompt.• Example: "My account ID is a zero five nine.." I2. Context-Triggered EntryEntity becomes relevant implicitly through conversational or external context.• Example: "I see order o nine five four from your previous case" ASSIGNMENT RULE:• Choose EXACTLY ONE initiation label based on how the entity first enters the conversation.• Focus on the trigger that made the entity relevant, not how it was articulated.RULE OF THUMB: Ask: "What happened IMMEDIATELY BEFORE the entity first appeared?"• → If there was a QUESTION or REQUEST for this entity → I0 (Elicited Entry) Key signals: "May I have..?", "Can you give me..?", "What is your..?", "Could you provide..?" • → If the speaker OFFERED the entity WITHOUT being asked → I1 (Volunteered Entry) Key signals: Speaker introduces entity mid-conversation, often with "My X is...", "I have...", "Let me give you..." • → If the entity was PULLED FROM CONTEXT (system, prior conversation, external source) → I2 (Context-Triggered Entry) Key signals: "I see...", "According to our records...", "From your previous case...", "The system shows..."IMPORTANT:• Ignore HOW the value was spoken (spelled out, fragmented, etc.) -that's Articulation.• Ignore WHETHER the value changed later.• Focus ONLY on the conversational trigger that brought the entity into discussion.
Authors: Soumya Jain, Ayush Kumar
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166807136