Year: 2026
Type: conference-paper
Abstract: High-quality instruction-tuning data is crucial for developing Large Language Models (LLMs) that can effectively navigate real-world tasks and follow human instructions.While synthetic data generation offers a scalable approach for creating such datasets, it imposes a quality ceiling where models trained on the data cannot outperform the LLM generating it.To overcome this limitation, we introduce REFERENCE-LEVEL FEEDBACK, a paradigm that extracts desirable characteristics from carefully curated reference samples to guide the synthesis of higher-quality instruction-response pairs.Using this approach, we synthesize REFED, a dataset of 10K instruction-response pairs.Fine-tuning Llama-3.1-8B-Instruct and Mistral-7B-Instruct on REFED demonstrate state-of-the-art performance among similarly sized models, notably reaching a 43.96% length-controlled winrate on AlpacaEval 2.0.Extensive experiments demonstrate that REFERENCE-LEVEL FEEDBACK consistently outperforms traditional sample-level feedback methods, generalizes across model architectures, and produces high-quality and diverse data at low cost.
Institution: University of Illinois Urbana-Champaign
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140106362