Get started
Work

Bipartite Concept-to-Token Language Modeling: Paraphrase-Invariant Mini Concept Planners with Context-Steered Token Expansion

Year: 2026
Type: preprint
Abstract: Autoregressive token-level Large Language Models (LLMs) conflate high-level semantic planning with low-level syntactic realization, expending O(L2) self-attention compute across surface tokens even when underlying clauses represent deterministic, paraphrase-invariant concepts. Conversely, pure sentence-level Large Concept Models (LCMs) compress en- tire clauses into continuous embeddings but lose exact syntactic and lexical steering at branching transitions. We propose the Bipartite Concept-to-Token Architecture (BCTA), which decouples language modeling into two specialized networks: (1) a Stage-1 Mini Concept Planner that operates on canonical, paraphrase-invariant concept codes (map- ping synonymous spans such as “sun rises in the east” and “sun rises in eastern direction” to a single quantized representation SRIE) interleaved with structural steering operators (<$>) to predict the next semantic state (SSIW) in O((L/k)2) complexity; and (2) a Stage- 2 Context-Steered Token Expansion Model that frames the planned sequence [SRIE, <$>, SSIW]into fluent subword tokens conditioned on global entity premises (e.g., <we have one sun> → “The sun rises in the east and that same sun sets in the west”) and a lightweight residual detail bypass that preserves exact numerical and lexical literals. We formulate the complete composite objective combining Paraphrase-Invariant Vector Quan- tization, Anti-Collapse InfoNCE, and Context-Contrastive Margin losses, differentiate BCTA from concurrent multiscale concept architectures, and detail the exact implementation blueprint for both a 15.2M-parameter Mini Prototype and a 3.8B-parameter Expanded Model.
DOI: 10.5281/zenodo.24140618
Language: English
Open Access status: green
OpenAlex ID: W7225772376
Cites:
Cited by:
Related to: