Year: 2024
Type: dissertation
Abstract: Test collections are a valuable source for evaluating systems in the information retrieval domain. They consist of a set of documents, topics and corresponding relevance labels, allowing the assessment of whether the output of a retrieval system for a specific topic includes the relevant documents. For popular research fields like text retrieval, plenty of large-scale test collections exist, while for less popular research fields, like table retrieval, only a few exist. Greater amounts of test collections can allow for broader evaluations, but limited resources and a lack of expert assessors hinder the creation of table retrieval test collections. With the rise of Large Language Models, new possibilities arise. This work is concerned with creating a table retrieval test collection by utilizing Large Language models to judge tables for relevancy on a range of topics. First, different prompts are exhibited, then a complete Web Table retrieval collection is judged by GPT-3.5. The results indicate that Large Language models like GPT-3.5 can be used to assist humans with table retrieval test collection creation. While plenty of tables are judged correctly, the portion of incorrect ones may be too large to automatically create a test collection without human verification. Still, there is great potential in significantly reducing the price and amount of work it takes to create such a collection.
Source: ePublication (TH Cologne)
Author: Grote, Nils
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7057696548