AECI: A Validation-First Benchmark for AI Economic Capability
Year: 2026
Type: article
Abstract: AECI: A Validation-First Benchmark for AI Economic Capability Engineering and Production Systems: From Specification to Verified Output Abstract The AI Economic Capability Index (AECI) is a proposed validation-first benchmark for evaluating whether artificial intelligence systems can complete economically meaningful tasks across real production workflows. Rather than evaluating AI systems solely through isolated questions, reasoning tasks, coding exercises, or the quality of individual outputs, AECI focuses on the ability to carry a task from an initial specification through design, coordination, verification, and delivery of an accepted output. The framework organizes evaluation by sector, business function, and concrete task. It examines several independent dimensions, including task completion, autonomy, embodiment, human intervention, cross-disciplinary coordination, production integrity, economic performance, and safety qualification. A central feature of AECI is the attribution of each workflow stage to its actual executor, distinguishing between AI systems, conventional software, machines, robots, and human operators. This makes it possible to distinguish genuine system capability from performance that depends on substantial human intervention. The proposed framework also evaluates the consistency of interconnected production artifacts, such as technical models, drawings, schematics, bills of materials, calculations, and operating instructions. A design package should not be considered successful merely because its individual documents appear plausible; the artifacts must remain mutually consistent and independently verifiable. AECI distinguishes three evaluation levels: T1 — Digital and simulation-based execution: Tasks performed using digital models, software, simulations, and verifiable technical artifacts. T2 — Controlled physical execution: Tasks involving controlled physical fabrication, assembly, installation, or machine-assisted execution. T3 — Real workplace or commercial operation: Tasks evaluated under actual operational or commercial conditions. As an initial engineering application, the proposal identifies a bounded mechanical, electrical, and plumbing (MEP) coordination task at T1. The proposed evaluation includes IFC geometry, clash detection, independent calculations, component catalogue selection, bill-of-materials generation, and cross-artifact consistency checks. The WEX-01 scenario provides a worked example, while additional held-out scenarios are proposed to reduce dependence on a single demonstration case. AECI is designed to complement existing AI evaluation initiatives by examining a different question: not simply whether an AI system can produce a convincing answer, but whether an AI-enabled production system can reliably transform a specification into a verified, economically meaningful output. Status of the Work This document presents a benchmark framework and validation protocol proposal. No experiments have been conducted, and no empirical performance results are claimed. Proposed thresholds and evaluation procedures remain provisional and require implementation and empirical validation. Keywords: Artificial Intelligence; AI Benchmark; Economic Capability; Production Systems; Engineering Automation; Autonomous Systems; MEP Coordination; IFC; Design Verification; Human Intervention; Workflow Automation; Cross-Artifact Integrity; AI Evaluation; Industrial Automation; Digital Production.
Author: Wibhawa, Atyanta
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225743507
FWCI:
Cites:
Cited by:
Related to: