Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) draw on both contextual information and parametric memory, yet these sources can conflict.Prior studies have largely examined this issue in contextual question answering, implicitly assuming that tasks should rely on the provided context, leaving unclear how LLMs behave when tasks require different types and degrees of knowledge utilization.We address this gap with a modelagnostic diagnostic framework that holds underlying knowledge constant while introducing controlled conflicts across tasks with varying knowledge demands.Experiments on representative open-weight and proprietary LLMs show that performance degradation under conflict is driven by both task-specific knowledge reliance and conflict plausibility; that strategies such as rationales or context reiteration increase context reliance, helping context-only tasks but harming those requiring parametric knowledge; and that these effects bias model-based evaluation, calling into question the reliability of LLMs as judges.Overall, our findings reveal that context-memory conflict is inherently taskdependent and motivate task-aware approaches to balancing context and memory in LLM deployment and evaluation. 1 4. Validation 3. Task-Annotation 2. Evidence Creation 1. Parametric Knowledge Collection Knowledge Collection TestLM Query the parametric knowledge with test LM.Curate conflict evidence & explanations Curate tasks using: -Knowledge Free -Contextual -Parametric -Combined Validate all data.Subsample for human reviews.
Authors: Kaiser Sun, Fan Bai, Mark Dredze
Institution: Johns Hopkins University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166824435