Year: 2024
Type: conference-paper
Abstract: Large Language Models (LLMs) have recently gained significant interest due to their impressive results in various natural language tasks.However, their application to sentence embeddings is still under active research.In this work, we introduce PromptEOL, a simple and efficient method designed to enhance LLM performance on sentence embeddings with a one-word limitation.We further integrate PromptEOL with in-context learning and alignment to leverage LLMs in two settings: without fine-tuning and with fine-tuning.Our extensive experiments show that PromptEOL enables LLMs to generate superior sentence embeddings without fine-tuning, outperforming contrastive learning methods.Additionally, with fine-tuning, a 2.7B parameter model using PromptEOL surpasses the performance of a 4.8B parameter model from previous methods.We also analyze how scaling model parameters, from 125 million to 66 billion, impacts sentence embedding performance.Our code and model is available at https:// github.com/kongds/scaling_sentemb.
Authors: Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang
Institution: Beihang University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G245388120, G6617559040
OpenAlex ID: W4404782013