GenTREC : The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems
Year: 2026
Type: article
Abstract: Building test collections for Information Retrieval evaluation has traditionally been a resource-intensive and time-consuming task due to reliance on manual relevance judgments. While various cost-effective strategies have been explored, the development of such collections remains a significant challenge. This article introduces GenTREC , the first test collection built entirely from documents generated by a Large Language Model (LLM), eliminating the need for manual relevance judgments. Assuming LLM-generated documents are inherently relevant to their prompts, we used TREC search topics to generate documents, considering a document relevant only to its generating prompt. To introduce realistic retrieval challenges, we also generated non-relevant documents, ensuring that IR systems are tested against a diverse and robust set of materials. The resulting GenTREC collection comprises 96,196 documents and 300 topics. We conducted extensive experiments to evaluate GenTREC in terms of document quality, relevance judgment accuracy, and evaluation reliability. Our results show that IR system rankings using GenTREC align with traditional TREC test collections for P@100, MAP, RPrec, and nDCG metrics. Our findings demonstrate that GenTREC offers a promising, low-cost alternative for IR evaluation, significantly reducing the burden of building and maintaining future IR evaluation resources.
DOI: 10.1145/3841465
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7203737890