Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation
Year: 2025
Type: conference-paper
Abstract: Large language models (LLMs) with instruction following capabilities have demonstrated impressive problem-solving abilities.While synthesizing instructional data from unsupervised text has become a common approach for training such models, conventional methods rely heavily on human effort for data annotation.Although existing automated synthesis paradigms have alleviated this constraint, they still exhibit significant limitations in ensuring adequate diversity and difficulty of synthesized instructions.To address these challenges, we propose SELF-FOVEATE, an innovative LLMdriven method for instruction synthesis.This approach introduces a "Micro-Scatter-Macro" multi-level foveation methodology that effectively guides the LLM to deeply excavate finegrained information embedded in unsupervised text, thereby enhancing both the diversity and difficulty of synthesized instructions.Comprehensive experiments across multiple unsupervised corpora and diverse model architectures validate the effectiveness and superiority of our proposed method.
Authors: Mingzhe Li, Xin Lu, Yanyan Zhao
Institution: Harbin Institute of Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: gold
Award: G2656429412
OpenAlex ID: W4412888402