Does Tool Output Truncation Cause LLM Agents to Answer Incorrectly? A Controlled Experiment with 1,198 LLM Calls
Year: 2026
Type: article
Abstract: Version 3 (20 Nov 2026). Corrects a table arithmetic error in Section 3.8: the pooled row reported 271 correct answers where the correct total is 274 (48.5% of 565 calls). The 271 figure is correct answers conditional on the answer being present; 3 further correct answers occurred when the answer had been truncated. The gap figure (95.7 percentage points) is unchanged. Version 2 (20 Nov 2026). Adds Section 3.8: replication on a real framework (LangChain BaseTool, end-to-end, no oracle), 565 LLM calls. P(correct | answer present) 96.8% [94.0; 98.3] versus P(correct | truncated) 1.1%; gap 95.7 percentage points, McNemar 83 down / 0 up, p = 2.07e-25. Adds five-seed bootstrap robustness (69.7 to 82.2 points on the oracle, 95.7 on the framework; leave-one-source-out worst case 72.3 points) and a framework source audit (OpenAI Agents SDK, LangChain, LlamaIndex truncate nothing by default). Question. arXiv:2610.36114 ("Context Is a Pipeline, Not a Window") measured that one third of tool results are truncated before reaching the model, and its authors left the causal question open: "Whether the data cut truly causes the accuracy drop is still an open question." Answer. This report answers that question with a controlled experiment over 1,198 real LLM calls, across two models (qwen3.6-flash and nvidia/nemotron-3-nano-omni-30b-a3b-reasoning) and two corpora (one self-authored, one public: DatNguyenAIML/ViLegalExperts on HuggingFace). Design. Oracle setup: the chunk containing the answer is always placed at position [1] of the context, so the only variable that changes is truncation length. Questions are cloze items that quote the source line with the target value masked as [X]. Results. Self-authored corpus, qwen3.6-flash (861 calls): 99.4% correct when the answer is present versus 23.3% when truncated. Gap 76.1 percentage points, z = 23.42, p < 1e-100. Cross-model replication, nemotron-30b (904 calls): 55.3% versus 7.3%. Gap 48.0 pp, McNemar 81 down / 4 up, p = 1.10e-19. Public corpus ViLegalExperts, nemotron-30b (1,198 calls): 67.7% versus 2.3%. Gap 65.4 pp. Confound check: of 87 correct answers despite truncation, 68 (78%) were answer leakage from another retrieved chunk. Removing the confound, true accuracy under missing information is 5.1% (19/373). Behaviour: when the answer is cut, models refuse in 67.3% of cases at a 150-character limit but only 25.0% at 1,200 characters; the remainder fabricate a value (37.8% overall). Framework audit: OpenAI Agents SDK, LangChain and LlamaIndex do not truncate tool output by default. Conclusion. Tool output truncation causes incorrect answers. Measured damage is 48 to 76 percentage points and scales with model capability: stronger models are damaged more. Recommendation. If truncation is unavoidable, cut from the beginning and preserve the tail (answers positioned late in the chunk survived at 40.4% versus 0.0% for answers at the start). Better: do not truncate, or use a recoverable compression scheme. Includes both an English report and a Vietnamese report. All data, code and results are retained; scripts are named in the appendix.
Author: 7EN Labs
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236793854
