GenTREC: A Synthetic Information Retrieval Test Collection
Year: 2026
Type: dataset
Abstract: GenTREC is a synthetic Information Retrieval (IR) test collection generated using Large Language Models based on standard TREC topics. The dataset is designed to assess the usability, reliability, and ranking compatability of fully synthetic document corpora in IR evaluation. By generating targeted subtopics and synthetic documents, GenTREC offers a fully controllable testbed. 1. Dataset Contents The package contains the following primary files: corpus.jsonl.gz: The synthetic document collection stored in compressed JSON Lines format. Each line represents an individual document structured as: id: Unique document identifier. contents: Complete document text exactly as generated by the LLM (including the document title at the beginning and the raw text format). qrels.txt: Complete relevance assessments in standard TREC format (topic_id 0 doc_id binary_relevance) 2. TREC Topics & Topic Access The synthetic documents in GenTREC are generated based on standardized TREC test topics: Topic Coverage: The collection covers 300 TREC topics: 251–300 (TREC-5 Ad Hoc track) 301–350 (TREC-6 Ad Hoc track) 351–400 (TREC-7 Ad Hoc track) 401–450 (TREC-8 Ad Hoc track) 601–700 (Robust2004 Ad Hoc track) Accessing Original Topics: To comply with NIST/TREC distribution guidelines, raw TREC topic definitions (title, description, narrative) are not directly redistributed within this archive. The original topic files can be accessed directly from the official NIST TREC website (https://trec.nist.gov). 3. Associated Publication (Please Cite) If you use this benchmark collection in your research, please cite our corresponding publication: @article{10.1145/3841465, author = {T{\"u}rkmen, Mehmet deniz and Kutlu, Mucahid and Altun, Bahadir and Cosgun, Gokalp}, title = {GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems}, year = {2026}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, issn = {1046-8188}, url = {https://doi.org/10.1145/3841465}, doi = {10.1145/3841465}, journal = {ACM Trans. Inf. Syst.}, month = aug, keywords = {Test Collection, Information Retrieval Evaluation, Relevance Judgments, Large Language Models}} 4. Repository & Project Information For dataset documentation, issue tracking, and metadata updates, please visit the GitHub repository: https://github.com/mdenizturkmen/GenTREC.git
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204194510