Work

Crawled but Not Indexed: A Corpus-Level Audit of Structural Homogeneity and Entity Sparsity on a 797-Page Corporate Website

Year: 2026
Type: preprint
Abstract: Generative engines answer brand queries by retrieving and citing web documents, and a growing literature on Generative Engine Optimization (GEO) prescribes what firms should publish. Far less is known about the prior failure mode: pages that are crawled, that carry valid structured data, and that nevertheless receive zero index coverage. We report a corpus-level audit of a production corporate website with 797 sitemap URLs, of which 91.7% belong to two templated sections. On a stratified sample of 100 pages we measure structural templating (tag-sequence n-gram Jaccard), content duplication (5-gram shingle Jaccard after boilerplate removal), structured-data coverage, and on-page hygiene. Results separate two phenomena that are usually conflated: structural homogeneity is extreme (mean Jaccard 0.652, p90 0.991, p99 1.000), whereas content duplication is low (mean 0.028, median 0.001). Every sampled page carries JSON-LD, a canonical URL, a unique title, a meta description and an H1; no page is marked noindex. Entity facts, however, are sparse: the ICP filing number appears on 100% of pages while the software-copyright registration number appears on 1%. The site had zero reported index coverage on the days of observation despite daily crawls. We argue that technical conformance and content originality are insufficient conditions for index inclusion at scale, propose a four-check diagnostic protocol separating crawlability, structural redundancy, entity completeness and content distinctiveness, and release the measurement script and per-page metrics to make the audit reproducible. 中文摘要 生成式引擎回答品牌类问题时,会先检索并引用网页。现有 GEO(生成式引擎优化)研究多讨论“该发什么内容”,却很少讨论更前置的失败情形:页面能被抓取、结构化数据齐全,却始终拿不到索引。 本文对一个拥有 797 条 sitemap URL 的企业站点做全量审计,其中 91.7% 的 URL 属于两个模板化栏目。我们在 100 页分层样本上分别测量结构模板化程度(DOM 标签序列 n-gram 的 Jaccard 相似度)、去模板后的内容重复度(5-token shingle Jaccard)、结构化数据覆盖与页面基础卫生。 结果把两件常被混为一谈的事分开了:结构同质化极强(均值 0.652,p90 0.991,p99 1.000),而内容重复度很低(均值 0.028,中位数 0.001)。全部样本页都有 JSON-LD、canonical、唯一标题、描述与 H1,没有一页被标 noindex。但实体事实稀疏:ICP 备案号出现在 100% 的页面正文,软件著作权登记号只出现在 1%。在观测期内该站每天被抓取,索引量却始终为 0。 据此我们提出四检诊断协议(可抓取性、结构冗余、实体完整度、内容独特性),并公开测量脚本与逐页指标,使审计可复现。
Language: English
Open Access status: green
OpenAlex ID: W7230185406
Cites:
Cited by:
Related to: 10