Year: 2026
Type: preprint
Abstract: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.
Source: arXiv (Cornell University)
Authors: Ghosh, Avijit, Reuel, Anka, Chim, Jenny, Kennedy, Wm. Matthew, Yadav, Srishti, Mickel, Jennifer, Long, Yanan, Tran, Andrew, Kornilova, Anastassia, Stachura, Damian, Klyman, Kevin, Friedrich, Felix, Sania, Jeba, Batzner, Jan, Mishra, Anoop, Habba, Eliya, Hao, Yixiong, Heath, Nathan, Rismani, Shalaleh, Gohar, Usman, Loehr, Andrea, Manheim, David, Dhar, Ruchira, Nelaturu, Sree Harsha, Sinha, Aarush, Choshen, Leshem, Sharma, Drishti, Khire, Ishan, Saha, Amit, Sahoo, Subramanyam, Hardy, Michael, Riegler, Michael Alexander, Manghnani, Kabir, Lin, Michelle, Jiang, Yanan, Huang, Yilin, Yehudai, Asaf, Ji, Jessica, Hofmann, Aris, Akhtar, Mubashara, Lamparth, Max, Moniz, Nuno, Jernite, Yacine, Biderman, Stella, Talat, Zeerak, Koyejo, Sanmi, Kochenderfer, Mykel, Solaiman, Irene
Institutions: Baden-Wuerttemberg Cooperative State University, Georgetown University, Stanford University, Queen Mary University of London, Technion – Israel Institute of Technology, Technische Universität Darmstadt, Harvard University, Weizenbaum Institute, Technical University of Munich, Amherst College, Hebrew University of Jerusalem, Iowa State University, McGill University, University of Copenhagen, Zuse Institute Berlin, IBM (United States), Massachusetts Institute of Technology, Georgia Institute of Technology, Mila - Quebec Artificial Intelligence Institute, Université de Montréal, ETH Zurich, University of Notre Dame, University of Edinburgh
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7164018997