Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) capture linguistic structure by operating on sequences of sub-word tokens, yet they often display behaviors that suggest an implicit grasp of high-level concepts.This study probes whether such "ideas" are genuinely encoded in LLM representations and, if so, how faithfully and to what extent.We created a deliberately minimalist LLM (encompassing both tokenizer and transformer architecture)designed to expose internal mechanisms with minimal architectural obscurity.Using a carefully curated toy corpora and probing tasks, we trace how semantically related prompts map onto the model's hidden states.Our findings reveal emergent clustering of conceptually similar inputs even in the stripped-down model.These insights advance our understanding of the representational geometry underpinning modern language models and outline a reproducible framework for future mechanistic studies of semantic abstraction.
DOI: 10.15439/2025f5480
Authors: Robert Chihaia, Maria Trocan, Florin Leon
Institution: Gheorghe Asachi Technical University of Iași
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Funder: Health Resources in Action
OpenAlex ID: W4415422008