DecideKV: a Free, Ready-to-Use Tool for Fast Typed Decisions with Any Local LLM
Year: 2026
Type: preprint
Abstract: DecideKV is a free, open-source Python library (MIT licence, https://github.com/HGarrone/decidekv) for a common job: given a state, pick one option from a list, give a level, or answer yes or no, with a local LLM. It is versatile and ready to use: one call, any GGUF model that llama.cpp runs (hybrid recurrent models included), lists from two to tens of thousands of options, and the Choice / Score / Noul question types of OpenJev. It works without any training, and it can be trained like CLM when examples exist, or without examples by letting the LLM label the states. Results, under equal conditions on one consumer GPU (same model, engine, prompt blocks and data): - 1.4 to 5.7 times faster than the OpenJev architecture, and at least as accurate in six of the seven task-model pairs compared (Banking77 with Qwen3-8B: 63.0% in 64 ms against 39.7% in 90 ms; WordNet-200 with a 27B hybrid model: 84.7% in 106 ms against 84.0% in 600 ms); - scales to 30,000 options, where OpenJev stops at 255; - with a catalogue trained in minutes: 93.0% on Banking77 in 15 ms and 75.6% on Typed Decisions with no LLM call, above CLM-8B with heads fine-tuned on the same tasks. How it works: the work is organised around the KV cache. Instructions, options and the question form one resident prefix computed once; every decision computes only the new state. Many questions about one state are answered as a prefix tree in one forward pass (a turn of 15 questions on a 27B hybrid model: from 11.2 s to 1.4 s with a long state). Above a few thousand options an embedding catalogue picks the candidates; the catalogue can be trained on human labels or on the LLM's own answers, with a fallback for options without examples and a confidence cascade back to the LLM. New in this version: the question goes into the cached prefix (never less accurate in paired tests on 9,580 decisions, significantly better on CLINC150); an optional semantic cache reuses the LLM's answers for near-duplicate messages; two recommended modes (structured states: the LLM with the prefix tree; short natural-language messages: a trained catalogue, or the LLM with the cache). Future work: with moving targets (interception in robotics, pedestrians in autonomous driving, demand in logistics, load in energy networks, games) the decision should be taken on where the target will be. The planned sequel puts a time-series forecaster in front of DecideKV: the forecaster passes the predicted state, and DecideKV decides on it with the same resident options and prefix tree. The paper also reports the weaknesses: where the tool is less accurate, how llama.cpp probabilities depend on batching and what was tried and did not work.
Author: Garrone, Hobey
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7235263160
