Evidence-access asymmetry, not scientific disagreement, drives inter-rater divergence in a blind reproducibility audit of passive daytime radiative cooling claims
Year: 2026
Type: preprint
Abstract: Background. Reproducibility assessment of published performance claims is increasingly proposed as an assurance service, but little is known about how reliably an independent second auditor can reproduce a first auditor's judgements, or what drives disagreement when they differ. Methods. Two auditors independently applied a frozen reproducibility protocol (RR2) to the same 50 passive daytime radiative cooling (PDRC) claims, deliberately sampled across publishers and evidence-availability regimes. The first auditor used a binary scale (reproducible / not); the second, working blind, used a three-way scale (PASS / FAIL / UNCERTAIN) with a recorded confidence. Decisions were reconciled without modification; disagreements were classified as access-driven, bibliographic, or substantive. The access-driven UNCERTAINs were then re-audited under enhanced evidence retrieval, and every claim's bibliographic identity was verified against authoritative metadata. Results. Exact three-way agreement was 19/50 (Cohen's κ = 0.26, descriptive). However, 28 of 31 disagreements arose because the blind auditor could not reach the evidence chain in its environment and returned UNCERTAIN, while only one disagreement was substantive and two were bibliographic. On the subset where both auditors could inspect the evidence, agreement was 19/20 (κ = 0.89, conditional). Re-auditing the 28 access-limited cases resolved 11 (39%), of which 10/11 aligned with the first auditor; effective agreement rose from 0.38 to 0.58, and resolvability depended strongly on access type. Bibliographic verification found and repaired 9/50 defective records, including one DOI that resolved to the wrong paper. Conclusion. Independent claim-level reproducibility assessment showed low apparent agreement under asymmetric evidence access, but agreement recovered substantially when the evidence became mutually inspectable. The observed reduction in agreement was predominantly associated with evidence-access asymmetry and label-space differences rather than with widespread substantive disagreement.
Author: Sánchez Lorenzo, Enrique
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7203888359