Human Oversight During AI-Assisted Documentary Coding: An Empirical Study of Decisions, Interventions, and Traceability
Year: 2026
Type: article
Abstract: A researcher agrees with an AI auditor that a coded category is unsupported and should be removed. The category is still present in the analysed dataset, and no record explains why. When decisions and data are recorded separately, such cases remain hidden, because neither record shows whether one reached the other. This retrospective study linked the two kinds of records in a completed project in which supervised AI systems coded 11,458 online reviews and 187 complaints about the Singapore–Malaysia confinement nanny market. Selected decisions were traced into the analysed data, and all 276 saved batch files were compared to identify changes that no decision explained. Whether a change could be traced depended on how it was made. Corrections logged by record and field were fully traceable. Recoding was not. Three recoded batches produced 109 changes, and only six of these could be linked to a decision about that record. Thirty-one saved files were also modified later, without their earlier contents being kept. Despite these gaps, none of the counts used in the project’s analysis changed, although only rebuilding them afterwards revealed this. A recorded decision is not evidence that it was carried out. The study proposes records that would provide that evidence, for future projects to test.
Source: Information
DOI: 10.3390/info17111177
Institutions: Multimedia University, University of Technology Malaysia
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,930
Funder: Multimedia University
OpenAlex ID: W7236856159
