AI-Readiness Evaluation of RO-Crates by LLM Rubric Grading: A FAIRSCAPE module for evidence-grounded scoring against the Bridge2AI AI-readiness criteria
Year: 2026
Type: article
Abstract: We describe a FAIRSCAPE module that evaluates the AI-readiness of biomedical datasets packaged as Research Object Crates (RO-Crates). The module scores each crate against the twenty-eight Bridge2AI AI-readiness criteria, organized along seven dimensions, using structured rubrics applied by a large language model (LLM). The criteria are evidence-of-conformance specifications: conformance is determined by what the metadata says, not whether the relevant fields are populated. An earlier FAIRSCAPE evaluator performs fast deterministic presence/absence checks against predetermined fields but cannot judge whether field content substantively addresses each criterion. Each rubric in the new module specifies an intent statement, the evidence to consider, a three-level ordinal scoring rule, and a JSON output schema (score, rationale, evidence citations, improvement gaps). For each criterion, a compact evidence payload is assembled from the candidate crate and supplied to the grader with the rubric. Per-criterion outputs are aggregated into a release-level report across the seven dimensions. The module was applied to a Bridge2AI Functional Genomics (CM4AI) multi-modal pre-release dataset as an initial case study. Corresponding author: Timothy W. Clark ([email protected])
Institution: University of Virginia
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7161574472
FWCI:
Cites:
Cited by:
Related to: