From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents
Year: 2026
Type: conference-paper
Abstract: Discovering improved policy optimization algorithms for language models remains a costly manual process requiring repeated mechanismlevel modification and validation.Unlike simple combinatorial code search, this problem requires searching over algorithmic mechanisms tightly coupled with training dynamics while reusing empirical evidence across iterations.We propose POISE, a closed-loop framework for automated discovery of policy optimization algorithms for language models.POISE maintains an archive that links each candidate design with its implementation, evaluation evidence, and analysis summary, enabling later iterations to build on both successful and failed designs.In mathematical reasoning experiments starting from GRPO, POISE evaluates 64 candidate algorithms and discovers improved mechanisms, including analytic-variance scaling and validity masking.The best variant improves weighted Overall from 47.8 to 52.5 (+4.6) and increases AIME25 pass@32 from 26.7% to 43.3%, demonstrating the feasibility of automated policy optimization discovery while supporting interpretable design principles.
Institution: Fudan University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7140405501