Work

Human Oversight During AI-Assisted Documentary Coding: An Empirical Study of Decisions, Interventions, and Traceability

Year: 2026
Type: article
Abstract: A researcher agrees with an AI auditor that a coded category is unsupported and should be removed. The category is still present in the analysed dataset, and no record explains why. When decisions and data are recorded separately, such cases remain hidden, because neither record shows whether one reached the other. This retrospective study linked the two kinds of records in a completed project in which supervised AI systems coded 11,458 online reviews and 187 complaints about the Singapore–Malaysia confinement nanny market. Selected decisions were traced into the analysed data, and all 276 saved batch files were compared to identify changes that no decision explained. Whether a change could be traced depended on how it was made. Corrections logged by record and field were fully traceable. Recoding was not. Three recoded batches produced 109 changes, and only six of these could be linked to a decision about that record. Thirty-one saved files were also modified later, without their earlier contents being kept. Despite these gaps, none of the counts used in the project’s analysis changed, although only rebuilding them afterwards revealed this. A recorded decision is not evidence that it was carried out. The study proposes records that would provide that evidence, for future projects to test.
Source: Information
Language: English
Open Access status: gold
Estimated APC paid: $1,930
OpenAlex ID: W7236856159
FWCI:
Cites: 22
Cited by:
Related to: 10