Healthy Spans, Failed Agents: Why Trace Sampling Loses the Evidence for LLM Agent Failures
Year: 2026
Type: preprint
Abstract: Trace sampling for distributed systems rests on an assumption: the span that explains a failure looks unusual on its own, so rules that keep errors, slow or costly spans, or structurally odd traces also keep the evidence. We show that this assumption does not hold for LLM agents, measure how much diagnostic evidence is lost as a result, and measure what it costs to keep it. On three public agent datasets, one recorded with real OpenTelemetry instrumentation, the span that annotators identify as decisive carries an ERROR status in only 9.8% of Who&When failures and 8.2% of TRAIL failures, and it usually sits several steps before any visible sign of trouble. At equal storage budgets, error, latency and cost rules keep failed runs at about the rate random sampling does, or worse, and anomaly samplers from the trace-sampling literature help only a little. Signals about the run’s outcome do better when they are informative: a metadata classifier keeps 2.6 times as many τ -bench failures as random sampling at a 15% budget, while an LLM judge helps on τ -bench but is no better than chance on TRAIL, where simply keeping the smallest traces retains more failures than any outcome signal we tested. The lost evidence matters: when a sampled view drops the decisive span, an LLM investigator finds the failing step far less often than from the full trace. Keeping that evidence has a systems cost. On the stock OpenTelemetry Collector, outcome-aware retention needs only a configuration change, but agent runs last minutes, so the decision window must too: with a 300 s window, 19% of flagged TRAIL runs are still lost, and collector memory grows with arrival rate, window and trace size. We release the converters, samplers, collector harness and configuration, all LLM outputs, and a onecommand reproduction script.
Author: Bisht, Kamal Singh
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7235044542
