Recall Is Not Execution: a Large Language Model and the Revised Geneva Score [Dataset].
Year: 2026
Type: dataset
Abstract: Study summary A frontier large language model (Claude Opus 4.8, model identifier claude-opus-4-8, Anthropic) was tested against the revised Geneva score for pulmonary embolism using simulated patient profiles. Two tasks were run. Task 1, weight elicitation. The model was asked to assign numeric weights to the components of the revised Geneva score, ten repetitions per prompt, under two independently worded prompts. Task 2, patient classification. The eight objective variables of the revised Geneva score were exhaustively enumerated to generate all 384 possible patient profiles. The model classified every profile as low, intermediate, or high probability of pulmonary embolism, under two wordings that describe identical patients and carry identical Geneva scores. Each wording was run twice, on separate dates, giving four classification runs in total. Analysis 3, internal coherence and negation density. Two further analyses were run on the classification data, without new model queries. The first tests whether the model's classifications satisfy the ordering that any risk rule with non-negative weights must satisfy: a profile carrying strictly more findings must not be assigned a lower probability category. The second tests, within strata of identical revised Geneva score, whether the number of findings stated as absent shifts the model's category. Both are model-against-itself comparisons and do not depend on the revised Geneva score being correct. The script pe_analysis.py reproduces both. No human subjects were involved. All patient profiles are synthetic combinations of the score's input variables.
Authors: Heston, Thomas F, Hwang, Grace
Institutions: University of Washington, Washington State University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7204471107