ChaosAthena: A Framework to Compare Gen AI Model States - A Post-Training Quantization Case Study
Year: 2026
Type: dissertation
Abstract: The Deployment of generative language models on resource-constrained edge devices depends on compression, and post-training quantization is among the principal methods.Its consequences, however, are assessed in silos: capability benchmarks, adversarial red-teaming, and theory-driven structural diagnostics run in isolation, so no single artifact records what a given deployable model state actually is.This dissertation addresses that gap through ChaosAthena, an extensible, hardware-portable evaluation framework integrating capability benchmarking, OWASP-mapped adversarial redteaming, and data-free Heavy-Tailed Self-Regularization (HTSR) and SETOL weight-matrix diagnostics as interchangeable Stages.A fully crossed case study covers 28 deployment-ready model states -seven open-weight families under eight billion parameters, each at FP16 and at three GGUF levels, probed by 39 distinct attacks.Four findings emerge under false-discovery-rate control.Capability degrades gradually and then collapses: pooled accuracy falls -0.6, -1.5 and -5.2 pp at 4-, 3-and 2-bit, concentrated in multi-step reasoning.The learned spectrum is preserved: the heavy-tail exponent α never leaves the well-trained band and is an architecture-family property (η 2 = 0.956).Safety separates into two structurally distinct axes: aggregate adversarial robustness is near-invariant, whereas intrinsic toxicity-safety degrades in 20 of 21 compressed states.Capability and adversarial safety are statistically decoupled (r ≈ -0.03, n.s.), so retained benchmark scores do not imply a retained safety profile.Furthermore, a set of twenty hypotheses, specified before their coefficients were computed, returns eight supported claims.Resolved per attack rather than per aggregate, weight structure indexes robustness in 26 of 36 testable probes, and the sign of each coupling is a property of the attack family rather than of the metric: jailbreak and injection probes invert one another on the shape metrics, while encoding and obfuscation attacks are indexed only by scale and richness quantities.Because opposing families cancel, a strong structure-vulnerability relationship can be invisible in an aggregate score.These results yield Pareto guidance identifying Q4_K_M as the preferred operating point and support the central claim: each compressed state must be evaluated as a distinct state in its own right, since trust is neither inherited from a model family nor inferred from a benchmark score.The same append-only schema that carries these results extends to directions the case study only begins to explore: structure-guided attack creation and synthesis, spectrally guided compression, data-free provenance screening, and evidence-grounded agent validation.
Author: Flávio Lobo Vaz
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7197441826