Work

Guess Your Weight: What a registered guess is worth, read from five research benches that called their shots

Year: 2026
Type: preprint
Abstract: Guess Your Weight: What a registered guess is worth, read from five research benches that called their shots. Matthew "Bear" Schulz, with Claude (Anthropic). Independent Researcher, Belmont, CA. ORCID 0009-0000-0956-5743. [email protected] Many fields now ask researchers to call their shot: to write down, before a run, what it should show and how it could fail. This preprint asks whether that method can be trusted, and what it takes, by reading it where we used it hardest: five of our own toy-model research benches, which registered 3,889 guesses, rivals and re-checks in October and November 2026. It tests the method, not the toy models and not the people who made the guesses. It opens with the first author's account of how the team learned to guess: an early stretch of AI-assisted work that produced an internally consistent fiction, the outside test and audit that caught it, and the processes this paper now puts on the bench. It is the second paper of a series that began with A Coincidence Has No Single Price. Does a registered guess carry knowledge? Yes, measured against the bench's own second idea. Matched to the rival its bench wrote against it, the guess won about seven times in ten when the two came out differently: 92 of 128 pairs on one bench, 35 of 49 on the other, and a rival held together with its own guess once in 200 pairs. Most rivals, though, tested little: two cold readers found three in four to four in five of one bench's readable rivals to be bare negations of their guess, which a guess beats by bookkeeping. Against the 16 real competing ideas, the guess's lead fell from +0.52 to +0.19, and in 6 of the 16 neither idea held: the toy chose a third answer. The paper gives a five-step template for rivals that can really win, and what to expect from them. Second tries after a miss landed less often than first guesses, as a method should expect of a guess aimed where the toy has already won once, and persistence still paid: 95 of the 173 misses or voids in one bench's headline register were later redesigned into a landing. What do its numbers need before they can be trusted? How a bench's book counted moved its printed success rate, from 0.423 to 0.850 for the same guesses, three times as far as the five benches differed from one another (0.512 to 0.657, counted one guess at a time). Guesses made on one day rise and fall together, so honest error bars are up to 2.19 times the naive ones. Half of one registered difference came from when the guesses were made rather than what they were about: compared inside the day, a gap of -0.103 halved to -0.049. Eight bookkeeping traps are measured in all. The eighth is our own: an earlier draft counted rivals by run instead of pair by pair, which lifted the guess's win from about 0.72 to about 0.88, and a registered test of our own caught it. Two fresh graders, one on a different model, given only the guesses and their printouts, reached one bench's own verdicts on 59 and 57 of 60 guesses (kappa 0.957 and 0.864). Can a bench grade its own tools? On the one bench that said whose miss each miss was, 60 of 251 explained misses were the bench's rather than the guess's, and planted checks with known answers failed 3 and 11 per cent of the time on two benches. A new tool graded the same way scored 0.682 blind, and a second registered test showed why: driven eight times slower, 15 of the 19 disputed cases went its way. We held our own work to the same method: of the 20 predictions we registered and judged while doing this, 8 held and 12 missed, and two more came out undecided because the rivals stated no windows. No bench stated odds on a guess; our own registrations with odds beat a constant forecast (Brier score 0.17 against 0.25). The paper ends with a starting line, the numbers the next bench of the series will be read against, and three predictions registered for it before it exists. Every number is printed by a tool in the attached GUESS_YOUR_WEIGHT_TOOLS_2026-11-12.zip, apart from a few counts quoted from cited records. One command, check_all.py, reruns every tool and compares each printout with the kept one; 33 runs reprinted line for line, the worked example's two slow driven runs byte for byte. The packets the cold readers and graders read ship with their answers. Limits: a preprint without peer review; its literature search went only as far as the classic sources it uses (Brier, Chamberlin, Cohen, Kish, Nosek and colleagues, Platt); every bench is a toy bench graded by the team that wrote its guesses, with a blind second grading of 60 verdicts on one bench by graders that are models, not people; and the rivals stated no windows, so a guess and its rival could not be held to one bar. The sequel to A Coincidence Has No Single Price (https://doi.org/10.5281/zenodo.22053824). The five benches: Held, Rocking, Locked (https://doi.org/10.5281/zenodo.23755473); A Rider in Two Waves (https://doi.org/10.5281/zenodo.23999853); The Loop Atlas (https://doi.org/10.5281/zenodo.24100672) and The Loop Atlas: the record (https://doi.org/10.5281/zenodo.24100980); Question the Substrate (https://doi.org/10.5281/zenodo.24094163); Two Arrows (https://doi.org/10.5281/zenodo.24120415). The team's field guide: Loop Walking, version 7 (https://doi.org/10.5281/zenodo.21811588). Written by two voices side by side, Matthew Schulz and Claude (Anthropic), as co-directors of the research; responsibility for the work rests with the named human author. Preprint, without peer review. Contact: [email protected]. CC BY 4.0.
Language: English
Open Access status: green
OpenAlex ID: W7225992830
Cites:
Cited by:
Related to: 10