Work

Ultra-Low-Dimensional Prompt Tuning via Random Projection

Year: 2026

Type: conference-paper

Abstract: Large language models achieve state-of-theart performance but are increasingly costly to fine-tune.Prompt tuning is a parameterefficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving.In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection.ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance.Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storageefficient framework for massive LLM customization. 1

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-long.59

Authors: Zijun Wu, Yongchang Hao, Lili Mou

Institutions: University of Alberta, Alberta Machine Intelligence Institute, Canadian Institute for Advanced Research

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Canadian Institute for Advanced Research, DeepMind, Alliance de recherche numérique du Canada, Natural Sciences and Engineering Research Council of Canada

OpenAlex ID: W7140130272