Work

Fetched, not followed: on one site, ClaudeBot and GPTBot repeatedly requested an llms.txt-format list and none of the 12 pages it listed

Year: 2026
Type: preprint
Abstract: We planted 36 static files on one website, each with exactly one intended route, and recorded which verified AI and search crawlers fetched them. Twelve pages were listed only inside an llms.txt-format file, announced only by its own robots.txt Sitemap line. Fetched, not followed. In the preregistered 21-day window, verified ClaudeBot requested the list 234 times and GPTBot 21 times. Neither fetched any of the 12 listed pages. Through 2026-11-17 the totals are 269 and 34, still 0 of 12. Other routes worked. The same crawlers fetched every file announced by its own robots.txt Sitemap line, and ClaudeBot fetched every file linked from the site footer. Google. Googlebot and GoogleOther followed a plain HTML link to a text file within hours. No Googlebot request was observed in 28 days to six plain text notes announced only by Sitemap lines (the notes were not valid sitemaps). Forecasts. Two of five preregistered predictions met their thresholds. What this is and is not. One site, one route for the list, a fetch log. It shows retrieval, not use, and it does not test llms.txt generally or the llms.txt filename. Two leading explanations remain, and the next design is built to separate them. Rival explanations are weighed with Heuer's Analysis of Competing Hypotheses. The preregistration was an internal document; its provenance, a mid-window logger change, the lost raw logs and the late reads are disclosed in full. Data. The deposit holds every logged crawler request to the study folder with classifier verdicts, and a script that re-derives every number in the note. The 12 control file names and the preregistration are withheld until after the 90-day re-read on 2027-01-12, to keep the control intact. Their SHA-256 hashes are in the deposit, and both files will be added as a new version of this record. Follows Hall (2026), Discovery, not adoption, doi:10.5281/zenodo.22814843.
Author: Hall, Nathan
Language: English
Keyword: robots.txt
Open Access status: green
OpenAlex ID: W7235429025
Cites:
Cited by:
Related to: 10