TraceForge: A Human-Gated System for Auto-Hardening Tool-Using LLM Agents into Immutable Workflow DAGs
Year: 2026
Type: conference-paper
Abstract: Tool-using language agents are flexible but nondeterministic: an orchestrator LLM re-plans on every request, adding latency, cost, and paths that are hard to audit.We demonstrate Trace-Forge, an on-premise system that observes how an agent actually solves a recurring task and, on operator request, hardens it into a deterministic, immutable workflow DAG.Trace-Forge attaches to the OpenTelemetry traces an agent already emits (zero code changes), reconstructs each run's execution DAG, and runs a human-gated loop: a statistical aggregation gate (proposes a DAG only once enough runs agree), a Judge-LLM reflection step, and a human freeze gate that writes a versioned, immutable specification with lineage and rollback.The pipeline is topology-agnostic-applied to single-agent tool-callers and multi-agent orchestrators alike-and all components run onpremise with no external dependencies.On held-out requests, frozen workflows reproduce the dynamic agent's outputs exactly while removing every planner-LLM call and cutting latency 1.39×.The entire loop is driven from a browser console with no coding, and is shown end-to-end in the accompanying screencast.
Author: Santhiya Rajan
Institution: Multiverse Computing (Spain)
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W7224296154