Talk Is Cheap, Show Me Your Colors: Low-Budget Consistency Auditing for Aggregated LLM Services
Year: 2026
Type: preprint
Abstract: Aggregated large language model (LLM) services expose declared model identifiers while hiding the backend configurations that fulfill client requests. We present HuePrint, a lightweight black-box framework that audits brokered endpoint consistency via structured color elicitation (#RRGGBB and R,G,B) and continuous geometric distribution comparison without model weights or token probabilities. Across 38,688 deduplicated live cloud responses spanning 56 endpoints and 12 backends (Datasets A–E and Studies S1–S5), HuePrint lowers 30-call-per-side equal error rates (EERs) over single-word baselines from 4.52–5.23% to 0.94–1.00% on the development cohort and from 6.25–8.04% to 3.49–5.00% across 20 locked endpoints, reaching 0.00% at 60–150 calls/side. In a same-day 20-model parity benchmark (Study S5), 30-call/side HuePrint averages 3.48% all-pairs (1.32–5.00%) and 5.18% within-family EER (0.00–7.89%; matched split 1.32% / 0.00%) across all ten balanced 3v3 splits at 0.0068 USD query cost, outperforming open-set LLMmap (10.53–34.39% within-family across 1–3 traces, while avoiding safety-triggering probes) and cost-matched long-form continuation kernels (11.09–19.47%), and scoring traces over 15,000 times faster locally (0.08–0.31 ms vs. 4,701.9 ms). Regularized RGB energy lowers shallow-census error over exact-hex divergence and string MMD (5.36% vs. 10.71% and 6.42%), and a multi-scale hybrid metric cuts cross-provider error across 12 hosts from 16.67% to 5.93%. Across 21-day cross-gateway checks, score separation stays exact (0.00% EER) and every rule catches 182/182 swaps, with budget-matched B6v6 frozen thresholds flagging 1/14 raw (0/14 self-centered) same-label pairs vs. 6/14 (42.86%) equality-test rejections. Unified Lipschitz bounds and estimator diagnostics guide calibration: self-centering lowers clean held-out false rejections from 30% to 10%, U-statistic energy ratios eliminate the 256-draw bootstrap, private prompt-pool subsampling cuts per-cell reuse by 4.0x, and two-stage confirmation reduces false alerts from 14 to 1 across 100 same-label windows while flagging 186/190 within-family substitutions under 0.01 USD expected query cost. Version 2.1.0 changes: Added the same-day head-to-head competitor benchmark Study S5 (4,160 new cloud responses across open-set LLMmap, Gao et al. long-form continuation MMD, and HuePrint; 38,688 unique responses total), operational monitoring analyses (private prompt-pool subsampling, two-stage sequential confirmation, Study S2 side-channel decontamination, external U-statistic verification, and all-10 balanced S5c split diagnostics), and refined manuscript prose across the 12-page IEEE Computer Society main paper and 20-page Supplementary Material. All prior 34,528 cloud responses and 81 SHA-256-locked baseline and live-validation artifacts remain bit-for-bit unchanged, joined by 10 newly SHA-256-locked Study S5 artifacts. This Zenodo record includes the complete preprint with supplementary material (HuePrint_Preprint.pdf, 32 pages) and a self-contained offline replication package (HuePrint_Replication_Package.zip) containing all 15 read-only JSONL response corpora, multilingual prompt batteries, statistical analysis scripts, unit tests, and SHA-256 integrity manifests.
Authors: He, Xiguan, Li, Siqi
Institution: Yunnan University of Traditional Chinese Medicine
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7235991739
