Work

Higher agreement, different errors: A preregistered out-of-sample evaluation of error-informed instructions for LLM coding of teacher talk moves

Year: 2026
Type: preprint
Abstract: Coding instructions for large language models (LLMs) are often revised from the models' errors against human codes, but rarely frozen and tested on new data, other models and single codes. In four preregistered studies on teacher talk moves, an optimizer model revised an LLM coder's instructions, and we tested the frozen result for transfer, costs and remaining errors. On locked test lessons, agreement with human codes (Cohen's kappa) rose from .67 to .76, below the preregistered target of .81 and not clearly above a revised manual. Unchanged, the instrument raised kappa by .08 to .22 on new teachers across five model families. Per move, kappa lay between .72 and .91, against .91 to 1.0 between the human annotators. Precision on rare moves rose for every coder, whereas their recall fell clearly for two coders. Higher kappa did not give more accurate move shares per teacher. In exploratory control runs, a generic instruction reached one fifth of the gain and rules written without human codes two fifths. Setting aside the 10% of sentences on which three models disagreed raised kappa on the rest to .92. Error-informed instructions transfer, but their errors depend on the coder and matter differently by use. Preprint. This version has not been peer reviewed. Supplementary Material S1, the software and all data and scripts are in the replication packages (https://doi.org/10.5281/zenodo.23911470).
Language: English
Open Access status: green
OpenAlex ID: W7236144741
Cites: 6
Cited by:
Related to: 10