Locating Information in Large Language Models via Random Matrix Theory
Year: 2024
Type: article
Abstract: As large language models (LLMs) become increasingly central to AI applications, understanding their inner workings is essential. In this work, we analyze the spectra of weight matrices in pretrained transformer models through the lens of random matrix theory (RMT) to uncover learned structures. We find that certain regions of the weight matrix spectra deviate markedly from RMT predictions, indicating richer feature encoding. By comparing the corresponding singular vectors to eigenvectors of activation covariance matrices, we observe substantial overlap precisely where the spectra deviate from RMT expectations. Our analysis further reveals the important role of small singular values in LLMs, showing that these values contain significant information—a claim supported by increased perplexity when they are removed from the model. Although these small values may appear unimportant prior to task-specific fine-tuning, removing them afterward significantly degrades performance, revealing that fine-tuning refines the model primarily in these spectral regions. These results emphasize the critical role of small singular values, suggesting that removing them in an already aligned transformer can be detrimental, as it may compromise model alignment.
Author: Anonymous
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6892959143
FWCI:
Cites:
Cited by:
Related to: