Work

Extending test collection pools without manual runs

Year: 2014
Type: conference-paper
Abstract: Information retrieval test collections traditionally use a combination of automatic and manual runs to create a pool of documents to be judged. The quality of the final judgments produced for a collection is a product of the variety across each of the runs submitted and the pool depth. In this work, we explore fully automated approaches to generating a pool. By combining a simple voting approach with machine learning from documents retrieved by automatic runs, we are able to identify a large portion of relevant documents that would normally only be found through manual runs. Our initial results are promising and can be extended in future studies to help test collection curators ensure proper judgment coverage is maintained across complete document collections.
Institution: RMIT University
Language: English
Open Access status: green
OpenAlex ID: W1990995988
FWCI: 1.052
Cites: 17
Cited by: 10
Related to: 10