Work

Reading Is Not Counting: A Three-Model Test of Context Dependence in Expert-Coded Invasion-Biology Studies

Year: 2026
Type: preprint
Abstract: Ecological studies can disagree because they test a hypothesis under different conditions. A literature synthesis must identify those conditions without inventing patterns. We test this task against Hi-Knowledge expert coding using 262 sets of eight studies, 299 study pairs and 386 single-study items from held-out papers. Claude Sonnet 5.5 (Anthropic, claude-sonnet-5-5), GPT-6.1 Sol (OpenAI, gpt-6.1-sol) and Muse Spark 1.3 (Meta, muse-spark-1.3-contributor) each receive the same items in five input forms for sets: title, abstract, model-made record, record with expert general conditions, and record with expert general conditions and link information. Pairs use three arms and single studies four. All use medium effort. The original design and a later model-extension plan were frozen before their respective test outputs. Scoring uses the expert coding, with no AI judge. With abstracts and an instruction to tabulate support by condition, Sonnet 5.5 finds 37/103 real patterns and raises 48/118 false alarms; GPT-6.1 Sol finds 58/103 with 99/118 false alarms; Muse Spark 1.3 finds 51/103 with 69/118. The false-alarm rate exceeds the detection rate in every model. Single-study abstract accuracy is similar: 278, 279 and 280 of 386. H2 favours expert conditions over model-made records in all three models, but is significant after Holm only for GPT-6.1 Sol and Muse Spark 1.3; no between-model effect test was run. GPT-6.1 Sol records lower detection (H1) and also lower false alarms (H4: 99 to 87/118; Holm p = 0.0338). Its single-study accuracy drop from 279 to 215/386 is descriptive, not the tested H1 loss. No pre-registered comparison is significant after Holm for Sonnet 5.5. The pattern remains visible in each model's own stances in 50/66, 18/45 and 32/52 abstract-arm misses. Exploratory code counting over those stances with expert conditions finds 83, 67 and 76 of 103 patterns, with 19, 16 and 24 false alarms. This is an upper bound using the scoring rule. The paired detection gain is significant for Sonnet 5.5 and Muse Spark 1.3 but not GPT-6.1 Sol, so the extension's planned all-model criterion is not met. Poor set-level reliability extends beyond the original model, but the balance of reading errors, missed patterns and false alarms differs. Evidence systems need separate checks on study reading, condition extraction and aggregation.
Language: English
Open Access status: green
OpenAlex ID: W7236715718
Cites:
Cited by:
Related to: 10