Work

Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation

Year: 2026
Type: article
Abstract: Background We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged. Objective We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates. Methods Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations. Results Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis. Conclusions Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.
Language: English
Open Access status: gold
Estimated APC paid: $3,077
OpenAlex ID: W7203981125
FWCI:
Cites: 34
Cited by:
Related to: 10
Parts: 1