Get started
Work

Carrying memory across a code-list revision breaks a precision promise: two public streams, a watch that notices, and a request for counterexamples

Year: 2026
Type: preprint
Abstract: A system that answers only the records it can answer at a stated precision, and leaves the rest to people, makes a promise: of the answers it gives, at least this share will be right. We report one way that promise fails and how quickly the failure can be seen. On two public record streams whose code list was revised so that code numbers were rewritten (Chicago food inspection violations, 2018 food code; OSHA severe injury reports, OIICS 3.0), a keeper that carried its memory over the revision promised 0.94 and 0.87 precision for the first 1,000 records after it and realized 0.44 and 0.007. A keeper that started its memory at the revision kept its promise in every range of records in which it answered. A watch that compares the last 100 answers whose rightness came back against the target raised its first alarm 54 and 163 records after the revision, with no alarm before it, and none on thirteen other public streams without a revision. Two score-threshold tools that freeze their threshold before the revision answered 92-100% of the records afterwards with fewer than 10% right. Everything here is two streams, derived (not human-confirmed) settlements, and one setting; a public kit rebuilds every table from the public data with a published package. We ask for counterexamples: a stream where carrying memory over such a revision keeps the promise, or where the watch stays silent after one. Replication kit: https://github.com/iyulab/gil-replication
DOI: 10.5281/zenodo.24069103
Author: iyulab
Language: English
Open Access status: green
OpenAlex ID: W7225662293
Cites:
Cited by:
Related to: