Structuring Neuralese: A Categorical Approach to Transformer Reasoning Beyond Attention
Year: 2025
Type: article
Abstract: We develop a rigorous categorical framework for analyzing transformer-based language mod- els, addressing the mathematical foundations that underpin their internal representations and computations. We prove that feedforward networks form an enriched category over augmented vector spaces, while attention mechanisms require 2-categorical structure due to their context- dependent nature. We then demonstrate how this theoretical foundation enables principled extensions to structured reasoning, proposing methods that preserve categorical structure while enabling more systematic reasoning processes. Our approach provides mathematical tools for understanding transformer architectures and suggests pathways for developing more structured reasoning algorithms.
Author: Janik, John
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6893276047
FWCI:
Cites:
Cited by:
Related to: