Boundary Screening Bounds the Evidence in Span Masking but Does Not Predict Linear-Probe Representation Quality at Small Scale
Year: 2026
Type: article
Abstract: Span masking is usually justified informally: contiguous masks are said to force phrase-level rather than token-level inference. We quantify that argument and then test it. A boundary-screening theorem shows that the evidence a context carries about a masked set passes entirely through the block boundaries, so it is bounded by a quantity scaling with the number of blocks rather than of masked tokens. The downstream prediction this motivates, pre-registered with fixed decision rules, is refuted under the tested design. Across a sixteen-fold range of span length at fixed masking ratio, probe accuracy varies by 0.80 points, against 2.0 to 4.3 point changes the same design resolves from auxiliary objectives and training length. Mask geometry therefore bounds the available evidence without predicting linear-probe representation quality at this scale. Two further predictions hold and a third in part: the Gaussian closed form matches direct computation to 10−13 nats, a radial conservation law for cosine objectives holds in a trained transformer to within 3 percent, and an auxiliary masked-language term degrades semantic probes while barely moving lexical ones. The theorem is exact for finite-order Markov chains; its natural-language counterpart is a character-level diagnostic, and the encoders are 4.5 M parameters trained on 74 M tokens, at which a random-initialised baseline is already strong.
DOI: 10.3390/make8110373
Authors: Suwichai Phunsa, Thawatchai Chomsiri
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,172
Funder: Mahasarakham University
OpenAlex ID: W7204990097