Year: 2026
Type: preprint
Abstract: A 143-second, captions-only screen recording of demo.agenticcrawl.ca, which replays recorded crawls of a generated test corpus. Agentic Crawl is a goal-driven web crawler: given a goal in plain words and the sites to read, it reads every start page, then always takes the queued link closest to the goal, judged by the link's own words and by the page it was found on. It decides every request before it is sent (the crawl's scope, the site's terms, robots.txt and its rules for AI crawlers), learns URL parameters that never change a page, stops by itself when the goal goes quiet, and writes every choice as a Brain API decision artifact. The recording follows one goal: the price and warranty of every tent an outdoor shop sells. A link to /cart is refused before sending because the shop's robots.txt disallows it, and the sites' own request log shows that two sites whose terms forbid the crawl, or are unclear, received no request at all. After three session-id copies, the crawler learns that the parameter sid never changes a page and skips such addresses; each row of the trace opens to its decision record and Brain API decision artifact. The crawl stops after 60 pages in a row well below its best, with 40 of 40 tent pages found. Held against the corpus's truth, 0 of 141 requests reached a page the rules forbid, 4 of 4 refusals gave the truth's reason, and all 40 pages were found in 90 requests, where breadth-first, on the same sites and rules, needed 1,121. A second goal, greenhouse tomato advice on a forum, found all 45 pages in 61 requests against 1,718. Scope. The sites are generated: nine sites and 1,799 pages, every page, link, rule, trap and change fixed before any crawl, served on one machine. No real site was crawled, and figures from this corpus are not figures for real sites. The demo replays the recordings; it crawls nothing and decides nothing in the browser. One recorded run per goal. Files: the video (1920x1080, H.264, no audio), a poster frame, and a transcript of the on-screen text.
Source: arXiv (Cornell University)
Author: Chernov, Alexander
Topic: Access Control and Trust
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7213914092
Cites:
Cited by:
Related to: