Get started
Work

SAS Multimetric Tribunal: What a Hallucination Benchmark Measures and What It Does Not — A Format-Controlled Audit of Structural Classification

Year: 2026
Type: preprint
Abstract: Hallucination-detection systems are usually judged by a single headline metric, computed under a benchmark design that determines what that metric can show. This paper’s contribution is not a detection algorithm — structural coherence scoring is established — but a methodology for measuring exactly what a benchmark proves and what it does not, applied to the SAS Multimetric Tribunal: a thirteen-module, GPU-free structural classifier combining lexical, entity-mutation, curvature, logical-inversion, topological-similarity, semantic-drift, entropy, complexity, and surprisal signals via a cascading multiplicative penalty (\(\kappa_{D} = 0.56\), \(\kappa_{R} = 0.15\)). On a stratified 1,800-pair benchmark (HaluEval-Dialogue, HaluEval-QA, TruthfulQA), the tribunal reaches F1 = 98.99% with zero false positives under an A\(\rightarrow\)A identical-text sanity check. We report this figure and then show, using a format-controlled negative (N1) on the same 1,800 IDs, that it measures format asymmetry between source and response rather than hallucination detection. The tribunal flags 1,569 of 1,800 verbatim correct answers presented alone (87.2%), and in HaluEval-QA scores the correct answer as _more_ anomalous than the hallucinated response. Three further findings follow from the same audit. (i) A per-module analysis shows the tribunal’s seven-module core has no content-sensitive module (prior to the optional R3.1 guard): the one module designed to capture semantic divergence (cre_isi) gives the same per-ID score whether the response is byte-identical or completely different. (ii) The 87.97-point advantage of multiplicative combination over weighted averaging reported in v4 is an artifact of threshold calibration; under per-scheme thresholds on format-matched negatives, both schemes converge to F1 \(\approx 0.69\) and the multiplicative scheme is worse by AUROC. (iii) A new differential guard (R3.1, grounding-based with compression abstention and short-response heuristics) corrects 1,133 of the 1,569 format-driven false positives on correct answers alone, raising F1 from 0.687 to 0.833 (\(\Delta\)F1 = \(+ 0.146\)), and is evaluated on a financial domain (FinQA, 1,657 samples) as a sanity check, with 87.0% detection and zero regressions on hallucinated answers (see §8.9 for framing). A held-out 50/50 split over 100 random repetitions confirms that the threshold (0.95) and F1 (0.833 \(\pm\) 0.0055 on held-out data) are stable across splits. The guard does not rely on substring matching and is not a constructional artifact in the sense of §8 (see §8 for the distinction between the guard’s design and the N1 construction). V9 ADDENDUM. Two post-peer-review advances are documented in this revision. (iv) The R3.1 guard’s binary compression-ratio abstention (CR \(>\) 0.5) has been replaced by a continuous weight function (\(\kappa\)C), reducing abstentions from 58 to 1 on N1 (\(-\)98%) without regressions. (v) A new topological-similarity module (R3.2) detects semantic equivalence under paraphrase when lexical overlap is near zero, using canonical clusters, deterministic lemmatization, passive-voice inversion, and a strict anchoring rule. R3.2 is integrated as an optional eighth module with a conditional gate (PARAPHRASE_RESCUE / CONTRADICTION_VETO / INACTIVE). Both advances were completed in approximately 10 hours total, following a specification-audit-code workflow. V9 ADDENDUM (2026-11-02). Four further extension modules are documented in this revision, bringing the tribunal to thirteen modules. (vi) R3.3 Semantic Drift Trajectory measures ISI sentence-by-sentence and localizes the break point where a response diverges from its source. (vii) R3.4 Entropy Differential compares Shannon entropy of source and response; on 300 calibration pairs the control condition yields \({|{\Delta H}|} = 0.000\) exactly, giving perfect separation. (viii) R3.5 Kolmogorov Complexity Ratio approximates complexity via zlib LZ77 compression. (ix) R3.6 Surprisal Analysis combines per-token self-information with an out-of-vocabulary fraction signal. All four modules are 100% standard-library, add no external dependencies, and were calibrated on real data rather than assumed thresholds. A full evaluation over 30,790 pairs across four public datasets (HaluEval-QA, HaluEval-Dialogue, TruthfulQA, MedHallu) yields aggregate precision 0.9989, recall 0.9877, F1 0.9933, and accuracy 0.9933, with 22 false positives in 20,768 control pairs (0.11%). The MedHallu corpus (10,000 records, 4-condition protocol) adds a format-controlled medical-domain evaluation with 85.10% detection and 0.01% identity false positives. V1.4.0 ANALYSIS SUITE (2026-11-05). Three further analyses are reported in this revision (§8.10). (x) A per-module firing breakdown across four conditions on the MedHallu corpus (10,000 records) shows that the lexical baseline (Jaccard) fires on 0.28% of format-controlled A\(\rightarrow\)A_′′_ pairs, confirming that detection is content-driven rather than length-driven. (xi) A structural fingerprint (12-dimensional feature vector: 3 relational + 9 absolute) achieves zero-overlap separation between paraphrase and hallucination: the 95th percentile of A\(\rightarrow\)A_′′_ (0.2400) lies below the 5th percentile of A\(\rightarrow\)B (0.5436), yielding a recommended rescue threshold \(\kappa_{P} = 0.4039\). (xii) Conditional compression (zlib-based \(\Delta_{SR}\) and asymmetry) is evaluated and reported as a negative result: both signals are dominated by length confounds, with A\(\rightarrow\)A_′_ asymmetry (647.9) indistinguishable from A\(\rightarrow\)B (670.6). The approach is discarded as a detection module. V1.4.1 FIX (2026-11-06). Two corrections are reported in this revision. (xiii) The R3.2 topological module’s dictionary-based gate was found to accidentally rescue hallucinations (127/1000 A\(\rightarrow\)B pairs) by matching common words to canonical clusters. Replacing the dictionary gate with the structural fingerprint distance (\(\kappa_{P} = 0.4039\)) eliminated all hallucination rescues, raising aggregate F1 from 0.8901 to 0.9933 (\(\Delta\)F1 = \(+ 0.1032\)). (xiv) Four fixes to the source_target_guard module (numerical magnitude comparison, drug name gazetteer, date mutation weight, causal inversion detection) raised adversarial detection from 1/5 to 5/5. Combined with the fingerprint rescue (5/5 on paraphrase cases), the adversarial set achieves 10/10. SCOPE. SAS detects structural hallucinations (content not present in the source) with coverage limited to texts with lexical overlap \(\geq\) 30% or shared entity structure. It does not detect: paraphrases with total restructuring and zero lexical overlap (requires distributed semantics), stealth hallucinations that preserve structure and entities (identical vocabulary with mutated facts), or false facts that do appear in the source. Declared coverage: 85.10% on format-controlled pairs (MedHallu, 10,000 records), with 0.01% identity false positives. After the v1.4.1 fixes, the adversarial set (10 manually constructed cases) achieves 10/10: 5/5 stealth hallucinations detected and 5/5 legitimate paraphrases rescued by the structural fingerprint (\(\kappa_{P} = 0.4039\)). The paper’s contribution is the audit procedure that produced these retirements. Five of v4/v5’s quantitative claims do not survive it. What does survive — thresholds, the \(\kappa_{R}\) sweep, the forensic audit of v2 metrics, the per-module architecture itself as a diagnostic instrument — is reported with the same standard.
Source: Qeios
DOI: 10.32388/t420iv.9
Language: English
Open Access status: diamond
OpenAlex ID: W7225628296
Cites: 6
Cited by:
Related to: