SpecSnap: Measuring Specification Retention and Regression Safety Across Multi-Session Agent Cutoffs
Year: 2026
Type: preprint
Abstract: Standard software engineering benchmarks evaluate coding agents inside a single uninterrupted conversation. In real engineering workflows, agents hit context window limits or session resets mid-sprint. SpecSnap tests what survives that handoff: Session 1 receives a multi-package specification in the conversation prompt, begins editing the repository, and is cut off immediately after its first .py source edit (before commit). A fresh session then starts in the same workspace with no prior conversation transcript and must finish the sprint. Across four synthetic repository tiers (70 total requirements scored across 733 held-out assertions), 91% to 100% of sprint specifications vanish in aggregate across the session cutoff:- Antigravity Gemini 4 Argon retains 6/70 (8.6% net | 7/70 raw)- Claude Code Opus 5.5 retains 4/70 (5.7% net | 4/70 raw)- Grok Build 4.7 retains 0/70 (0.0% net | 0/70 raw) To prevent memorization, a seeded generator varies identifiers, thresholds, formulas, and prompt phrasings across more than 211 billion combinations while keeping pytest grading deterministic. SpecSnap also includes a locked Stage II evaluation (held-out state perturbation) across Tiers 1-4 gated at PASS (100% held-out tests passing and 0 AST boundary violations) plus >= 75.0% Token-Efficiency Score (TES). Because no evaluated frontier model has reached the 75.0% TES threshold on any tier (peaking at 50.0% TES on Tier 1 and 12.5% TES on Tiers 2-4), Stage II remains locked. Live Leaderboard: https://specsnap.devInteractive Whitepaper: https://whitepaper.specsnap.devOpen-Source Benchmark Repository: https://github.com/adamm285-dev/SpecSnap
Author: Murphy, Adam
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225586425