Self-Healing JSON Generation in Small Language Models: A Systematic Evaluation of Syntax Repair, Schema Coercion, and Validator Feedback
Year: 2026
Type: preprint
Abstract: Small language models offer an affordable way to extract structured information, including on devices with limited computing resources. However, their responses often contain JSON that cannot be parsed or does not satisfy the required schema. We investigate a self-healing approach that operates on generated text alone, without changing or accessing the model's decoder. The approach combines three steps: (L1) a tolerant lexer and bracket-stack parser for syntax repair, (L2) schema-guided conversions that preserve the original meaning, and (L3) a retry mechanism that provides JSON Schema validation errors to the model. We test these steps on 120 seeded text-to-JSON tasks spanning four schema families, using three instruction-tuned models ranging from 360M to 1.5B parameters and six experimental conditions. On average, only 16.7% of raw responses satisfy the schema. Applying L1 and L2 raises validity to 67.8% without another model call. The complete pipeline achieves 78.1% validity with an average of 1.57 calls per task. The results reveal several important distinctions. Qwen2.5 frequently produces correct JSON inside Markdown code fences, making many failures easy to correct without involving the model again. SmolLM2-360M, by contrast, generally avoids code fences but often misunderstands the requested structure or copies the schema itself; validator feedback is more useful in these cases. Feedback on its own performs poorly (21.1% validity), largely because it does not break persistent formatting habits. Compared with blind resampling at a similar call budget, feedback improves results for the weakest model (p = 0.003), although the overall difference is not statistically significant (p = 0.060). More importantly, producing valid JSON does not guarantee accurate information: 45.6% of valid self-healed responses still contain at least one incorrect field, or 38.4% under a more forgiving comparison. We observe empty strings, fabricated phone numbers, and default enum choices among these errors.
Source: Research Square
Author: Ahmad Abdelrahman
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225643282