Cognometry v0.5: Three Calibrated Cognometric Instruments (Hallucination, Refusal, Tool-Call Drift) + Phase-Transition Ablation
Year: 2026
Type: preprint
Abstract: Cognometry v0.5 — extends v0 with two additional calibrated instruments (refusal, tool-call drift) and a phase-transition ablation result shipped in the same release window (2026-04-23). Three calibrated cognometric instruments: - Hallucination: 0.998 AUC on HaluEval-QA, 8-benchmark cross-validated (v4) - Refusal: 0.976 AUC on XSTest-v2 GPT-4 held-out, 18-feature LR trained on Llama-3.2-1B apologetic refusals; empirical cross-substrate universality confirmation (v5.1) - Tool-call drift: 0.916 AUC on BFCL v3 5-fold CV, 22-feature LR, text-only. Beats the only published comparable baseline (Healy et al. 2026, AUC 0.72 on Glaive, hidden-state MLP) while being black-box compatible (v6.0) Phase-transition addendum: failure-class detectability does NOT rise smoothly with feature count. Each drift class has a critical feature that flips it from chance to near-perfect. arg_drop 0.50 -> 0.998 at K=2 (+arg_count_zscore); arg_swap 0.48 -> 0.69 at K=6 (+type_mismatch_frac); irrelevance_called 0.83 -> 0.96 at K=10 (+prompt_coverage). Inverse of emergent-capabilities literature for generative LLMs. Software: pip install styxx==6.0.0 Code: https://github.com/fathom-lab/styxx (MIT on code, CC-BY-4.0 on weights) Reproducers: 3 min CPU each, no API, no LLM inference Playgrounds: https://fathom.darkflobi.com/cognometry (Pyodide, no install) OSF bundle: https://osf.io/6syq4/ Tests: 655 passing, every AUC number backed by a committed script
Source: OSF Preprints (OSF Preprints)
Authors: Rodabaugh, Alexander, Flobi
Institution: Fathom (United Kingdom)
Language: English
Topic: Mental Health via Writing
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7155383922