Concept Decodability Across Chess Architectures: Testing Whether Human-Aligned Models Encode Concepts More Linearly
Year: 2026
Type: conference-paper
Abstract: Introduction: Recent work has pushed toward human-aligned chess engines capable of predicting moves at a specific skill level.Maia-2 [1], a hybrid CNN-transformer, and Allie [2], a GPT-2-style transformer, exemplify this shift: both are trained on human games rather than self-play, unlike superhuman engines such as Stockfish [3].Prior work shows that superhuman models also encode linearly interpretable concepts in chess [4].This raised the question of whether concepts are more decodable in human-aligned models.We test this directly by comparing concept decodability across two human-aligned architectures (Maia-2, Allie) and one purely evaluation-driven architecture (Stockfish's NNUE), asking whether human-alignment predicts the extent to which concepts are decodable.Methodology: Using an 81-concept binary battery across six concept groups (king safety, tactical forks, tactical pins, positional structure, pawn structure, material), with a synthetic dataset of 10,000 positive and 10,000 negative entries per concept, we extract activations via forward hooks and train linear support vector machine probes at each model's pre-output layer, our anchor point for crossarchitecture comparability.This is in accordance with prior work showing this layer gives consistent, comparable results in both CNN and transformer models [5].Decodability is scored via F1 under stratified 60/20/20 splits.Statistical significance is assessed with a repeated-measures framework treating concepts as the unit of analysis, first across all 81 concepts jointly, then separately within each of the six semantic groups, using a Friedman test with Kendall's W for effect size, followed by Holm-corrected Wilcoxon signedrank tests for pairwise comparisons.Results: As Figure 1 shows, across nearly all concept groups, Maia-2 achieves the highest decodability and Allie the lowest, with Stockfish (SF) intermediate.However, the size of this architecture gap, and the variance within each model, differ by concept.Maia-2 was most decodable in king safety (F1 = 0.772), tactical forks (0.794), tactical pins (0.723), positional structure (0.829), and pawn structure (0.717).Material was the only category where Stockfish outperformed (0.733), though this difference was not statistically significant due to low power.Allie decoded concepts less linearly than both other architectures.The Friedman test confirmed the overall differences were significant (χ²(2) = 99.56,p < .001)with high concordance in ordering (Kendall's W = 0.615), and Holm-corrected pairwise comparisons showed Maia-2 significantly ahead of both other architectures on king safety, tactical pins, positional structure, and pawn structure.This offers the first systematic, statistically grounded map of shared versus architecture-specific concept representation in chess-playing networks.Conclusion: Human-interpretable chess concepts are decodable across fundamentally different architectures, but Maia-2 encodes concepts most linearly at the pre-output layer among the three, making it the strongest candidate for translating internal representations into explanations within the intelligent tutoring system, at least for the concept classes it captures best.Future work will extend this analysis using causal validation via activation patching and sparse-autoencoder-based feature discovery, toward a skill-adaptive intelligent tutoring system for chess.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225447100