Year: 2025
Type: conference-paper
Abstract: Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision.We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses.Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions.For deeper analysis, we focus on neurons that encode distinct semantic concepts.By tracing their development during pre-training, we show how they gradually align across languages.Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.This alignment manifests concretely in generation: once an MLLM exhibits cross-lingual generalization according to our measures, we can select concept-specific neurons identified from, e.g., Spanish text and manipulate them to guide token predictions.Remarkably, rather than generating Spanish text, the model produces semantically coherent English text.This demonstrates that cross-lingually aligned neurons encode generalized semantic representations, independent of the original language encoding.
Authors: Frederick Riemenschneider, Anette Frank
Institution: Heidelberg University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945246