Work

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

Year: 2026

Type: preprint

Abstract: Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research at https://github.com/rangehow/mtr-suite.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2605.20729

Authors: Ruan, Junhao, Abudula, Abudukeyumu, Li, Bei, Yin, Yongjing, Liu, Xinyu, Jiao, Kechen, Chen, Xin, Wang, Jingang, Cai, Xunliang, Xiao, Tong, Zhu, Jingbo

Institutions: Meituan (China), Northeastern University, Tsinghua University

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7161936626

Cites:

Cited by:

Related to: