Eigen-Spectral Probe: Evaluating Weight Matrix Spectra and Cumulative Energy Retention in SLMs
Year: 2026
Type: other
Abstract: This spectral audit presents an empirical examination of weight matrix geometries, singular value distributions, and cumulative energy retention across the SmolLM2 architecture family (135M, 360M, and 1.7B parameters). Operating entirely without reliance on theoretical specification sheets or assumed scaling laws, the diagnostics inspects raw state dictionaries directly in memory. By executing rigorous Singular Value Decomposition across every attention projection, feed-forward block, and embedding tensor, the framework captures the true underlying spectral characteristics of sub-billion parameter transformers under local CPU execution. Core Empirical Findings Layer-Wise Spectral Decomposition: SVD execution across query, key, value, output, and FFN projection matrices demonstrates that matrix spectra do not follow a clean power-law decay (R2 values ranging from 0.63 to 0.89), but instead exhibit distinct structural elbows. Effective Rank and Energy Concentration (k_90): Quantitative profiling measures the exact fraction of components required to retain 90 percent of spectral energy. While FFN matrices demonstrate broader dispersion, the embedding layer ($W_E$) in SmolLM2-135M shows extreme concentration with its top-1 singular value carrying 50.6% of total energy. Cross-Scale Invariance Collapse: Scaling from 135M to 1.7B parameters reveals that spectral properties do not remain constant; W_E top-1 concentration drops sharply from 0.506 to 0.093, proving that spectral defaults are strictly model-specific. Truncation Fidelity vs. Frobenius Error: Empirical evaluation confirms that Frobenius error does not reliably predict output drift. Truncation at k_90 preserves output fidelity (KL < 0.05), though CPU deployment testing indicates that pruning smaller sub-billion tensors yields minimal wall-clock speedup due to memory bandwidth limits. Structure & Reproducibility The repository comprises six self-contained diagnostic notebooks and modules: 1_spectra_real_weights.ipynb: SVD execution and spectral metrics across actual weight tensors. 2_semantic_directions.ipynb: Projection of token embeddings into spectral coordinates and frequency regression. 3_truncation_fidelity.ipynb: Rank truncation versus output drift and KL divergence. 4_scaleout_spectra.ipynb: Comparative spectral scaling across 135M, 360M, and 1.7B variants. 5_weight_probing.ipynb: Linear probing from spectral components to semantic attributes. 6_spectral_pruning.ipynb: Deployment-time spectral pruning and performance benchmarking. Built for complete reproducibility on standard hardware, all diagnostic routines rely on pinned dependencies (torch, numpy, scipy, matplotlib) and execute cleanly in local, CPU-only environments—ensuring that every spectral claim can be independently audited, verified, and built upon without specialized infrastructure.
Author: kayani, Zeeshan
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7214482016
