Get started
Work

Balanced Defensive Fine-Tuning for Text-to-SQL on a Small Code Language Model: A Pre-Registered Measurement Report

Year: 2026
Type: preprint
Abstract: Locally run small language models enable generating SQL from natural language without sending the database schema to third-party servers. However, a model fine-tuned only to "translate the question into SQL" can also translate destructive or unauthorized commands embedded in the prompt. In this report we measure, under a pre-registered protocol, the Balanced SFT method applied with QLoRA (nf4, r=16) on Qwen2.5-Coder-3B-Instruct. In this method each training example is drawn from the attack set with probability α and from the legitimate Spider set with probability 1−α. The compared conditions are: the untrained base model (B0), standard SFT on legitimate data only (A0, α=0), balanced SFT (A.3, α=0.3), and SFT on defense data only (A1, α=1.0). Each trained condition was run with 3 seeds. The test sets consist of 500 attack prompts, 1034 Spider-dev questions, and 24 legitimate questions that contain risky words. Attack success rate (ASR) measured with a database-execution (DB-exec) judge is 61.53% ± 2.53 on A0 and 0.00% on A.3. The difference is −61.53 points, with 95% confidence interval [−64.73; −58.53]. Official test-suite execution accuracy (EX) is 70.25% ± 0.49 on A0 and 69.50% ± 0.88 on A.3. The difference is −0.74 points, with 95% confidence interval [−2.06; 0.55]. Thus H1 (security) and H2 (cost, non-inferiority) pass the pre-specified thresholds. However, the protocol-mandated human judge audit of 100 examples was not performed and will not be performed (Appendix A, S18). ASR was measured only with the automatic DB-exec judge; because judge validity was not tested against human labels, both hypotheses are reported only as supported (provisional), and this label can no longer become definitive. The over-refusal hypothesis H3 remains exploratory because the test set was reduced from 250 to 24: even with 0/24 errors the Wilson upper bound is 13.8% and lies above the 10% threshold. The α=1.0 condition collapsed on all three seeds to a solution that rejects every input. Deviations from the locked plan and unplanned analysis decisions are collected in a single table in Appendix A, with dates and whether they were taken before or after results were seen (11 coded deviations, 6 uncoded decisions; 17 items, 18 rows). Seven of these items were taken after results were seen. Among them are the 14 October 2026 finalization of the decision to stop at 3 seeds, the merging of the category 3/4 rule in the DB-exec judge, and moving the human-audit target from A.3_s11 to A0_s11. The collapse of A1 and the fact that ASR is a proxy measurement are not deviations but a result and a measurement note.
DOI: 10.5281/zenodo.23373230
Language: English
Open Access status: green
OpenAlex ID: W7221752650
Cites:
Cited by:
Related to: