Work

Can Language Models Recognize Reused Evidence? Cross-Task Errors and Prompt Sensitivity in Provenance Judgments

Year: 2026
Type: preprint
Abstract: Agreement among reports need not imply independent underlying evidence. We evaluate whether four open-weight instruction-tuned models (1.5–3.8B parameters) distinguish established source independence (YES), confirmed reuse (NO), and unresolved provenance (UNCERTAIN), and whether global judgments agree with local reused-pair identification. On a 24-family, 72-case constructed narrative corpus, confirmed-reuse recall under the archived P0 instruction is 0–3.47%, despite higher aggregate and pair-task accuracies. Three frozen prompt variants on the same narratives produce pronounced answer-distribution shifts: Phi-3.5 reaches 75% NO recall with explicit definitions but responds NO on every input when the permitted answer order changes. On a distinct, pre-inference-frozen 36-family structured-register corpus (6,048 GPU responses), global macro-F1 ranges from 0.184 to 0.404; Qwen2.5-1.5B incorrectly asserts independence on 256/432 designated reuse cases, whereas SmolLM2’s 100% reuse recall comes from near-constant NO predictions. Across both constructions, constant-answer baselines, permutation-invariant errors, and global–local disagreement make aggregate accuracy an unreliable stand-alone measure. These are controlled diagnostics against author-assigned labels, not estimates for unrestricted scientific papers. A separately supplied independent single-human-reviewer record covers 96 cases but cannot yet be aligned case by case with the frozen corpora. Retrospective lexical-shortcut susceptibility also limits inference from narrative scores.
Language: English
Open Access status: green
OpenAlex ID: W7236789591
Cites:
Cited by:
Related to: 10