ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
Year: 2025
Type: conference-paper
Abstract: Conversational search systems require effective handling of context-dependent queries that often contain ambiguity, omission, and coreference.Conversational Query Reformulation (CQR) addresses this challenge by transforming these queries into self-contained forms suitable for off-the-shelf retrievers.However, existing CQR approaches suffer from two critical constraints: high dependency on costly external supervision from human annotations or large language models, and insufficient alignment between the rewriting model and downstream retrievers.We present ConvSearch-R1, the first self-driven framework that completely eliminates dependency on external rewrite supervision by leveraging reinforcement learning to optimize reformulation directly through retrieval signals.Our novel two-stage approach combines Self-Driven Policy Warm-Up to address the cold-start problem through retrievalguided self-distillation, followed by Retrieval-Guided Reinforcement Learning with a specially designed rank-incentive reward shaping mechanism that addresses the sparsity issue in conventional retrieval metrics.Extensive experiments on TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly outperforms previous state-of-the-art methods, achieving over 10% improvement on the challenging TopiOCQA dataset while using smaller 3B parameter models without any external supervision.
Authors: Changtai Zhu, Siyin Wang, Ruijun Feng, Kai Song, Xipeng Qiu
Institutions: Fudan University, UNSW Sydney, ByteDance (China)
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G2565984771, G8590833815
OpenAlex ID: W4416035553