Year: 2026
Type: conference-paper
Abstract: Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval failures confound subsequent steps.Standard outcome-based optimization only rewards the final answer, leaving intermediate retrieval and reasoning errors undetected.While existing process-based methods introduce step-level signals, they still score each step against the final answer, rewarding spurious successes where flawed retrieval coincidentally produces the correct answer.Steplevel supervision in RAG requires evaluating both logical validity and evidential grounding at each step.We introduce PRO-STEP: we train a generative PRM that evaluates both dimensions, employ PRM-guided value tree search to construct preference pairs contrasting valid steps against flawed ones, and optimize the policy via step-level Direct Preference Optimization.Experiments on singleand multi-hop QA datasets demonstrate that PRO-STEP achieves the best average EM and F1 across five benchmarks.Code, models, and training data are publicly available at https://github.com/keemminnke/PRO-Step.
Authors: MinKeon Kim, Namjun Lee, Jaekwang Kim
Institution: Sungkyunkwan University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7207541998