Year: 2017
Type: dissertation
Abstract: High-recall retrieval is an information retrieval task model where the goal is to \nidentify, for human consumption, all, or as many as practicable, documents relevant to \na particular information need. \nThis thesis investigates the ways in which one can evaluate high-recall retrieval \nsystems and explores several design considerations that should be accounted for when designing \nsuch systems for electronic discovery. \n \nThe primary contribution of this work is a framework for conducting high-recall retrieval \nexperimentation in a controlled and repeatable way. \nThis framework builds upon lessons learned from similar tasks to facilitate the use \nof retrieval systems on collections that cannot be distributed due to the sensitivity \nor privacy of the material contained within. \nAccordingly, a Web API is used to distribute document collections, \ninformations needs, and corresponding relevance assessments in a one-document-at-a-time manner. \nValidation is conducted through the successful deployment of this architecture in the 2015 TREC \nTotal Recall track over the live Web and in controlled environments. \n \nUsing the runs submitted to the Total Recall track and other test collections, we explore the \nefficacy of a variety of new and existing effectiveness measures to high-recall retrieval tasks. \nWe find that summarizing the trade-off between recall and the effort required to attain that \nrecall is a non-trivial task and that several measures are sensitive to properties of the test \ncollections themselves. \nWe conclude that the gain curve, a de facto standard, and variants of the gain curve are the most robust to \nvariations in test collection properties and the evaluation of high-recall systems. \n \nThis thesis also explores the effect that non-authoritative, surrogate assessors can have \nwhen training machine learning algorithms. \nContrary to popular thought, we find that surrogate assessors appear to be inferior \nto authoritative assessors due to differences of opinion rather than innate inferiority in \ntheir ability to identify relevance. \nFurthermore, we show that several techniques for diversifying and liberalizing a surrogate \nassessor's conception of relevance can yield substantial improvement in the surrogate \nand, in some cases, rival the authority. \n \nFinally, we present the results of a user study conducted to investigate the effect that \nthree archetypal high-recall retrieval systems have on judging behaviour. \nCompared to using random and uncertainty sampling, selecting documents for training using relevance sampling significantly decreases the probability that \na user will identify that document as relevant. \nOn the other hand, no substantial difference between the test conditions is observed in the time taken to render \nsuch assessments.
Source: UWSpace (University of Waterloo)
Author: Adam Roegiest
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
OpenAlex ID: W2592079337