Evaluating Biomedical AI Co-Scientist Workflows: A Factorial Study of Proposal Quality, Assessability, and Resource Use
Year: 2026
Type: article
Abstract: AI co-scientist systems combine literature retrieval, hypothesis generation, critique, and refinement through language-model agents. Google's Co-Scientist and ResearchAgent support scientific proposal development; Google's system has also reported experimental follow-up in acute myeloid leukemia drug repurposing and liver-fibrosis target discovery. How workflow choices affect evaluation scores and computational requirements remains incompletely characterized. We evaluated an architecture adapted from Google's AI co-scientist using 76 retrospective biomedical questions, fixed pre-2024 evidence, and time-restricted PubMed retrieval. Five binary factors yielded 32 configurations plus two references, each evaluated in three replicates, producing 7,752 GPT-6 Luna outputs. GPT-6 Astra separately graded proposal quality against supplied literature evidence and outcome agreement against later published findings; the grader performed no searches. Across 228 complete blocks, retrieval was associated with higher top-three mean rubric scores (+3.32 points; 95% case-bootstrap interval, 2.66 to 4.02) and rank-one scores (+4.00; 3.23 to 4.82), while additional descendants were associated with lower rank-one scores (−3.79; −4.28 to −3.31). Retrieval required approximately 100,622 additional selected-attempt tokens per assignment. No clear retrieval-score advantage was detected in a later common-evidence regrading of 19 complete questions. This comparison changed both evaluation evidence and grading occasion, so it did not isolate an evidence effect. This check did not assess outcome agreement. Repeated grading also showed score variability. These findings identify grading-protocol sensitivity and judge variability as central considerations when assessing co-scientist configurations. Independent biomedical validity, selective outcome assessability, execution-period effects, and residual training-data contamination remain unresolved. AI-reviewed by the Journal of Autonomous Scholarship. Authenticated submitting researcher: 0000-0002-6249-7193. AI disclosure: See manuscript Data/code: See manuscript Funding and conflicts: See manuscript Ethics: See manuscript
Authors: Ruheng Wang, Wen Fan, Xiaowei Zhan
Institution: The University of Texas Southwestern Medical Center
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225566072