MENTOR: An Evidence-Informed Large Language Model-Based Multi-Agent System for Efficient Selection of Institution-Specific Negative Control Outcomes
Year: 2026
Type: preprint
Abstract: Objective Negative control outcomes (NCOs) help detect and correct systematic error in observational studies, yet their selection remains largely manual. The Common Evidence Model (CEM) can generate candidate lists, but researchers must still screen them for plausibility as true nulls and for sufficient observation in each institution's data. We developed and evaluated MENTOR (Multi-agent Evidence-informed Negative conTrol Outcome selection with Reasoning), a multi-agent system of large language models (LLMs) that combines CEM evidence with LLM reasoning to select institution-specific NCOs. Methods Four LLM agents operate on an institution's Observational Medical Outcomes Partnership common data model. A Curator summarizes drug labels, a Suggester assesses locally frequent candidates with CEM evidence and a first-pass LLM judgment, an Adjudicator resolves their disagreements, and a Diagnostician relaxes the patient-count threshold until a target number of NCOs is met. Using Qwen3.6-27B, we evaluated 349 candidates for three drug pairs in a Korean tertiary hospital's electronic health records (2.9 million patients) against a blinded LLM judge from a different model family, validated by three experts on 90 candidates. We compared accuracy, positive predictive value (PPV), and sensitivity with CEM-only and first-pass-LLM-only selection. Results MENTOR proposed 243 NCOs versus 121 by CEM-only, with higher accuracy (0.83 [0.79-0.87] vs. 0.53 [0.48-0.58]), sensitivity (0.89 vs. 0.41), and PPV (0.86 vs. 0.79). Judge-expert agreement was κ=0.63. Adjudication, needed for 44.1% of candidates, added little accuracy, and only with reasoning mode. Concordant acceptances, which bypass it, contained about half of the false positives. Only 7%-9% of CEM-proposed outcomes had at least 10,000 patients at the institution. Conclusion Combining LLM judgments with CEM evidence and local data yielded more NCOs, with higher PPV, than CEM alone. Restricting reasoning to discordant candidates kept computation time below 40% of that with reasoning in all agents. Every decision comes with its evidence and rationale for investigators' final review.
Source: medRxiv
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225885830
