Work

Does Benchmark Critique Diffuse into Evaluation Practice? A Pilot-Based Study of EvalPlus and SWE-bench+ Adoption

Year: 2026
Type: preprint
Abstract: This record is a pilot-study preprint together with its LaTeX source and reproducibility materials. It reports a small, purposive study of whether research papers that cite a benchmark critique go on to use the remediation that critique released. Two remediations are examined. EvalPlus augments HumanEval and MBPP with additional tests; SWE-bench+ releases a filtered 548-instance dataset for SWE-bench. A frozen citation frame of 2,666 title clusters was built from Semantic Scholar records retrieved on 13 November 2026. From that frame, 29 papers were drawn by purposive sampling: 27 full texts were accessible, 12 papers were coded, 8 were eligible for the all-pair set, and 6 for the two primary pairs. Rubric v1.1 distinguishes citation only (0), acknowledgment (1), partial adoption (2), and full adoption (3). One of the six primary-pair papers fully adopted the remediation (16.7%; Wilson 95% CI 3.0–56.4); the all-pair figure is 1 of 8 (12.5%; 2.2–47.1). Three of six primary-pair papers, and four of eight all-pair papers, remediated independently instead of adopting the critique's artifact. The agreement statistics reported in the paper are LLM–LLM (κ = 0.78 unweighted, 0.86 linear weighted) and are not human inter-rater reliability. Status. The paper is a pilot study and a preprint. It has not been peer reviewed and has not been accepted for publication. It does not estimate prevalence in the 2,666-cluster frame, and it does not establish causal diffusion from critique to practice. The full-scale study described in the protocol — a stratified sample of 900 invited papers, human double-coding, and pre-specified sensitivity analyses — was designed but not executed. Contents. paper.pdf (20 pages, A4); paper-source.zip (XeLaTeX source, vector figures and figure data, abstract, reference metadata); reproducibility-package.zip (coding rubric, pilot coding table, analysis scripts and tests, generated tables, protocol, provenance and licensing notes, aggregate frame summary); README.md; LICENSE; LICENSE-DATA. Licensing. The paper, documentation, rubric, and derived data are released under CC BY 4.0. Original code is released under the MIT license. The frozen citation frame and the arXiv metadata export are not redistributed here because they derive from Semantic Scholar records; aggregate counts and SHA-256 hashes are included, and the frame itself is available in the repository under Semantic Scholar's terms. Links. Repository: github.com/P-r-e-m-i-u-m/benchmark-critique-diffusion-protocol. Research website: p-r-e-m-i-u-m.github.io/benchmark-critique-diffusion-protocol.
Language: English
Open Access status: green
OpenAlex ID: W7233608541
Cites:
Cited by:
Related to: