Информационное поле объектов в больших языковых моделях: от архитектуры к формированию представлений
Year: 2026
Type: article
Abstract: we reviewed the architectural principles and algorithms of modern generative neural networks and large language models. We examined the core mechanisms of text processing, including tokenization, embeddings, deep contextualization, autoregressive generation, and context management. We analyzed the transformer architecture in detail and emphasized self-attention as the key innovation that enables parallel sequence processing and captures long-range dependencies efficiently. We described common architecture variants (encoder-only, decoder-only, and encoder–decoder), positional encoding methods, and the computational complexity of major components.We studied pretraining on large text corpora and subsequent adaptation through fine-tuning and instruction tuning. We examined alignment with human preferences using reinforcement learning from human feedback (RLHF) and reviewed efficient adaptation methods, including low-rank adaptation (LoRA) and quantization. We introduced the concept of an object information context, which we define as the overall amount and distribution of information about a given object in digital data, including all references to that object in the training corpus. We showed that the structure of this context in the training data influences the formation of object representations in the model’s embedding space. We also examined the quality of the data environment and showed that low-quality sources degrade model accuracy, and we discussed approaches to mitigate these effects.
Authors: Бондарева, Н. А., Бондарев, А. Е.
Institution: Keldysh Institute of Applied Mathematics
Language: Russian
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7220151502