Work

Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study

Year: 2025

Type: conference-paper

Abstract: Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. While the CLTR models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate, their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely available, large-scale, real click logs. However, many previous simulation-based experiments are somewhat limited because they may have one or more of the following deficiencies: 1) using a weak production ranker to generate initial ranked lists, 2) relying on a simplified user simulation model to simulate user clicks, and 3) generating a fixed number of synthetic click logs. As a result, the robustness of CLTR models in complex and diverse situations is largely unknown and needs further investigation.

Source: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

DOI: 10.1145/3726302.3730310

Authors: Zechun Niu, Zhilin Zhang, Jiaxin Mao, Qingyao Ai, Ji-Rong Wen

Institutions: Renmin University of China, Tsinghua University

Language: English

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: National Natural Science Foundation of China, Renmin University of China, Fundamental Research Funds for the Central Universities

Award: G6620384876