Get started
Work

Hayula Neural Transformer: Multi-Scale Residual Stream Architecture with Global Workspace Gating for Security Language Models

Year: 2026
Type: article
Abstract: We introduce the Hayula Neural Transformer (HNT), a novel architecture that bakes Global Workspace theory (Baars, 1988) into transformer design through Multi-Scale Residual Streams. Inspired by the discovery that J-lens vectors form a global workspace in standard transformers—with broadcast heads specialized for relaying workspace content and MLP layers preferentially amplifying workspace-aligned directions (Gurnee et al., 2026)—HNT explicitly separates computation into three residual streams: a main stream for standard autoregressive computation, a local stream for syntax and surface-level features concentrated in early layers, and a global stream for workspace-level conceptual content concentrated in middle layers. A Cross-Stream Gating (CSG) mechanism, parameterized as a low-rank bilinear form inspired by QK-circuit analysis (Anthropic, 2025a), controls information flow between streams while minimizing feature interference. We further introduce Broadcast Heads: attention heads whose OV circuits are optimized for workspace content relay using gain-based selection metrics derived from MLP amplification analysis. Applied to security-domain training on a curated corpus of threat intelligence, vulnerability descriptions, and secure code repositories, HNT achieves 18% greater concept depth (measured by J-lens projection magnitude) than equivalent-parameter standard transformers, with a 22% reduction in feature interference as measured by the interference weight metric (Anthropic, 2025b). Our results demonstrate that architectural specialization informed by mechanistic interpretability yields disproportionate gains in domain-specific reasoning tasks.
DOI: 10.5281/zenodo.21537192
Author: Yahya Saqban
Language: English
Open Access status: green
OpenAlex ID: W7170707752
FWCI:
Cites:
Cited by:
Related to: