Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models
Year: 2025
Type: preprint
Abstract: Large language models (LLMs) excel at generating contextually relevant content. However, tailoring these outputs to individual users for effective personalization is a significant challenge. While rich user-specific information often exists as pre-existing user representations, such as embeddings learned from preferences or behaviors, current methods to leverage these for LLM personalization typically require costly fine-tuning or token-heavy prompting. We propose Embedding-to-Prefix (E2P), a parameter-efficient method that injects pre-computed context embeddings into an LLM's hidden representation space through a learned projection to a single soft token prefix. This enables effective personalization while keeping the backbone model frozen and avoiding expensive adaptation techniques. We evaluate E2P across two public datasets and in a production setting: dialogue personalization on Persona-Chat, contextual headline generation on PENS, and large-scale personalization for music and podcast consumption. Results show that E2P preserves contextual signals and achieves strong performance with minimal computational overhead, offering a scalable, efficient solution for contextualizing generative AI systems.
Source: arXiv (Cornell University)
Authors: Huber, Bernd, Fazelnia, Ghazal, Damianou, Andreas, Peleato, Sebastian, Lefarov, Max, Ravichandran, Praveen, De Nadai, Marco, Lalmas-Roellke, Mounia, Bennett, Paul N.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6966818366
Cites:
Cited by:
Related to: