Semantic Atoms for Language Models: Hierarchical Diffusion over Attention-Invariant Embeddings
Year: 2025
Type: article
Abstract: This research proposal introduces a novel framework for enhancing reasoning and semantic understanding in large language models through three interconnected innovations: Attention-Invariance Scoring (AIS): A method to extract stable semantic primitives ("atoms of meaning") from context-dependent embeddings by analyzing embedding variance across diverse contexts. Hierarchical Diffusion for Semantic Recognition: An adaptation of diffusion models for document understanding rather than generation, learning to compress million-token documents into multi-scale semantic abstractions. Content-Adaptive Semantic Allocation: Inspired by Gaussian splatting techniques from computer graphics, this approach allocates representational capacity based on information density rather than uniform segmentation. The framework addresses fundamental limitations in current language models: lack of compositional structure, inefficient long-context processing, and brittleness in novel reasoning tasks. By separating semantic reasoning (operating on stable concept atoms) from surface-level generation, we propose a pathway toward models that "think in ideas" rather than tokens. Status: This is a pre-experimental research proposal and conceptual framework. The ideas are presented with proposed experimental protocols, but empirical validation remains as future work. Comments, feedback, and collaboration inquiries are welcome. Keywords: Large Language Models, Semantic Compositionality, Diffusion Models, Hierarchical Abstraction, Long-Context Understanding, Attention Mechanisms
Author: Ataman, Mehmet Zeki
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7084592728
FWCI:
Cites:
Cited by:
Related to: