Routing, Summarization, and Memory in Neural Architectures For Reasoning and Language
Year: 2026
Type: dissertation
Abstract: Scaling neural networks through larger models and longer context has enabled substantial improvements in their capabilities but has also dramatically increased their computational and memory costs. This dissertation develops methods for enabling efficient scaling using sparse models, context summarization and pruning, and neural memory. For model scaling, we study Mixture-of-Experts, where sparsity via routing allows scaling model size without a proportional increase in compute. We identify a limitation of standard Top-K routing, where unselected experts provide no gradients to the router, and introduce \emph{Default MoE} which provides a gradient signal for the unselected experts to learn better routing. For context scaling, we first study whether intermediate computations can be pruned from a reasoning LLM's context. \emph{Structured Thoughts} organizes reasoning into exploratory reasoning and their summaries. By training models to continue reasoning from summaries, we find that the exploratory reasoning can be pruned from the context, allowing models to perform extended reasoning while maintaining a shorter context. The second approach to efficient context scaling uses a memory mechanism with attention-based reads and writes, providing a flexible mechanism for storing and retrieving context. Our memory learns to solve several synthetic tasks requiring context storage and, when paired with local self-attention, enables language modeling that approaches the performance of full-attention Transformers in both pretraining and finetuning. We further investigate a hierarchical memory design with sparse access as a path towards scaling memory capacity efficiently.
Source: University of Chicago
DOI: 10.6082/007tj-8yn91
Author: Sarwar, Zain
Institution: University of Chicago
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222049030
