PALO BlueprintBench: An Evidence-Grounded Methodology for Evaluating Specification Fidelity in AI-Generated Business Blueprints
Year: 2026
Type: article
Abstract: AI-generated business specifications can preserve the vocab-ulary of a source document while changing an obligation, permission, exception, or decision authority. Evaluating this transformation requires separating syntactic conformity, se-mantic fidelity, execution reliability, and the human work needed to repair an output. We present PALO Blueprint-Bench, an open benchmark module and evaluation method-ology for this problem. The method binds source scope, raw outputs, proposition annotations, configurations, and accep-tance policies to inspectable evidence. It distinguishes crit-ical source-rule inversions, unsupported critical additions, required-proposition coverage, and correction effort, without collapsing them into an unvalidated composite score. We re-port four diagnostic evidence cohorts: historical adapter tests on three passages containing 4,994 UTF-8 bytes; an inter-nal procurement pilot with 18 model configurations, 324 as-signed calls, and three synthetic cases from one authored fam-ily; a six-scenario feasibility cycle with 48 provisional propo-sitions and 60 requests to three configurations; and a real-document generation study on three public procurement doc-uments totalling 200 pages, in which 14 of 27 assigned calls returned a contract-valid specification.
Author: Degni, Fabrizio
Institution: Politecnico di Milano
Language: English
Subfield: Law
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225894550
