Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Year: 2023
Type: conference-paper
Abstract: The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns.High-performing open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance.To address this gap, we propose (1) A untargeted variant of iterative selfcritique and self-refinement devoid of external influence.(2) A novel ranking metric -Performance, Refinement, and Inference Cost Score (PeRFICS) -to find the optimal model for a given task considering refined performance and cost.Our experiments show that SoTA open source models of varying sizes from 7B -65B, on average, improve 8.2% from their baseline performance.Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks on the Vicuna benchmark.Vicuna-13B takes it a step further and outperforms ChatGPT postrefinement.This work has profound implications for resource-constrained and informationsensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy.The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by case studies.
Institution: University of Illinois Urbana-Champaign
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4389518896