Year: 2025
Type: article
Abstract: Large Language Models (LLMs) have transformed natural language processing, achieving state-of-the-art results in text generation, reasoning, and problem-solving.Despite these advances, aligning LLM outputs with nuanced human preferences remains a challenge, hindered by the inefficiencies and instability of traditional reinforcement learning (RL) methods such as Proximal Policy Optimization (PPO).These multi-stage pipelines often introduce high computational costs and degrade core model capabilities.This paper proposes two unified RL-based algorithms, Odds Ratio Preference Optimization (ORPO) and Group Relative Policy Optimization (GRPO) which combine supervised fine-tuning (SFT) and preference alignment into a single training phase.This integrated approach eliminates the need for separate reward models and sequential stages, significantly reducing the risk of catastrophic forgetting while enhancing training efficiency.Empirical evaluations on Mistral-7B and Llama-3-8B across six benchmarks (MMLU, MATH, GSM8K, HumanEval, BIG-bench, and TruthfulQA) show that ORPO outperforms PPO, achieving a 23% improvement in reasoning tasks and a 37% reduction in training time.Lyapunov-based theoretical From Complexity to Clarity: One-Step Preference Optimization for High-Performance LLMs https://iaeme.com/Home/journal/IJAIML
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4409686497