Found but not picked: How code-graph context loses to BM25 in small Gemma 4 windows
Year: 2026
Type: preprint
Abstract: Context: Bellam (2026) reports that code-graph tools add 17 to 21 percentage points of strict function Acc@10 over grep alone for 1.7B to 4B models, at a 32k-token budget. Objective: Test whether code-graph context still helps small Gemma 4 models localize issues in small windows, through three research questions: whether it beats BM25 code, whether model size or repo-to-window ratio changes its gain, and why it loses. Method: On 766 issues from 25 Python repositories, fixed after Gemma 4's cutoff, we give four Gemma 4 sizes a flat file tree, BM25 code, or one of two graph layouts at 2k and 8k tokens. Results: BM25 scores above both graph layouts in all 8 model × budget cells, narrowly for 26B at 2k (41.3% vs 40.7%). The graph names the faulty function at least as often as BM25 (coverage 53.7% vs 47.5% at 2k), but Gemma 4 picks it less often (E2B pick rate: 48.4% vs 72.5%): found, but not picked. Running the graph from BM25's seeds closes 3.7 percentage points of BM25's Acc@1 lead on E2B at 2k and 2.3 on E4B at 8k. Each hop from seed to faulty function multiplies the odds by 0.55 to 0.65, and graph layouts deepen that penalty at every size. Conclusion: In small windows the graph finds the faulty function, but Gemma 4 picks it less often. Code and data (Apache-2.0) are in the attached zip. Preprint of the manuscript submitted to Information and Software Technology (Elsevier). An earlier, non-archival version was entered in the Kaggle Gemma 4 Developer Agent Paper Track (2026).
Author: Kandukuri, Tharun Reddy
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225993623
