Work

Measuring the Indus corpus before testing it: edition disagreement, near-copy leakage and detection limits

Year: 2026
Type: preprint
Abstract: Statistical and machine-learning work on the Indus script rests on a corpus of 2,270 distinct short texts. We measure three properties of this corpus that decide what such work can show. (1) The editions disagree. Matching 469 objects across the two standard transcriptions through the 1930s excavation registers, independently of their signs, we find that the editions read 12.9% of signs differently (95% CI 11.1–14.9%; 14.0% out of sample), and one text in three differs somewhere. (2) Random splits leak. Because moulded tablets and fixed formulae put near-copies of a text on different objects, a random split by object leaves a near-copy of 37% of test texts in training even after exact repeats are removed (median of 200 splits; 95% range 33–42%), and of 62% if they are not. (3) Many questions lie below the detection limit. Planted-answer controls show that a genre difference must exceed 0.57 bits per text to be found, that a test for a relational reading of a joining sign fails at Indus size and works on a corpus four times larger, and that a sign-order statistic passes on four of seven non-linguistic codes. We release a family-level split and a corrected concordance between the two sign lists, and recommend reporting practice for future quantitative work on the script.
Language: English
Open Access status: green
OpenAlex ID: W7236602298
Cites:
Cited by:
Related to: 10