Work

Evaluating the Impact of Automated Labeling on Retrieval Instability in Neural IR

Year: 2025

Type: conference-paper

Abstract: Effective information retrieval (IR) depends on accurate relevance classification. But when the criteria are subjective or underspecified, small variations in classification can cause consequential shifts in retrieval results. The potential for such variability becomes critical for institutions when they use IR for research assessment. Retrieval instability can lead to relevant literature being overlooked, hindering a comprehensive understanding of the research landscape, and potentially undermining the validity of subsequent analyses and decisions.

Source: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

DOI: 10.1145/3726302.3730128

Author: William A. Ingram

Institution: Virginia Tech

Language: English

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funder: Universitas Brawijaya

Award: G4917478585