Measuring a Local AI Companion Through Its Own Runtime: Benchmark Defects, Run-to-Run Noise, Model Scale and Prompt Layout in B-12/Capri
Year: 2026
Type: preprint
Abstract: A previous report described B-12, a local-first agent runtime, and Capri, its desktop companion. This follow-up asks what an evaluation of such a system actually measures. We ran a 92-scenario suite (80 runnable on the available hardware) through the companion's own runtime inside a filesystem sandbox, instead of calling the language model directly. The first evaluation, which bypassed the runtime, scored 27 of 80 and produced a privacy leak that the harness itself had created; the same suite run through the runtime scored 39 of 80 before any change to the companion. Auditing the benchmark exposed further measurement errors: an incorrect reference value that failed two models for a correct answer, a symbolic-link test whose fixture was an ordinary text file, and scenarios that typed marker strings instead of exercising the mechanism they named. With these corrected, three repeated runs of each configuration gave 49, 49 and 48 passes for the primary 4-billion-parameter model (Nemotron Bolt E4B with Gemma 4 12B for swappable scenarios), 45, 43 and 47 for Gemma 4 26B (mixture of experts) and 45, 45 and 42 for Gemma 4 31B. Totals hid the noise: between two identical runs, three to ten scenario verdicts changed. Most of the gap between configurations came from latency bounds measured on different hosts; ignoring time bounds, means were 49.7, 48.0 and 47.0, and a paired comparison by scenario found no difference (p = 0.85). Sixteen scenarios failed in every run of every configuration even without time bounds, which locates the main limits in the system and the benchmark rather than in model scale. Two interventions illustrate how small prompt decisions dominate a small model's behaviour. A single explicit brevity sentence halved conversational reply length, but work tasks started with a tool call less often (32 to 19 of 48 first turns) and passed less often (32 to 21 of 48). Moving recalled memories from the system text to a message after the user's turn let the inference server reuse its prompt cache, cutting median time to first token from 0.99 to 0.74 seconds without a detected aggregate change in scenario success, although two scenarios shifted systematically. Results come from a consumer laptop and, for the larger-model comparisons, a cloud GPU, on an author-built benchmark; we report them as an engineering case study with raw data. This preprint continues B-12 / Capri: A Local-First AI Companion with Traceable Memory and Controlled Tool Use (doi:10.5281/zenodo.24079168). Code, scenario suite, raw scenario-level results, reply texts, experiment scripts and the script that computes every reported number are in the B-12 repository (paper/capri-evaluation-2026-11-14).
Author: Pylypenko, Oleksandr
Institution: State University of Trade and Economics
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7233600711
Cites:
Cited by:
Related to:
