When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models
Year: 2026
Type: preprint
Abstract: Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers' handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement.
Source: arXiv (Cornell University)
Authors: Li, Yubo, Krishnan, Ramayya, Padman, Rema
Institution: Carnegie Mellon University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
Awards: G1453181110, Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, TITLE: CMU/NIST AI MEASUREMENT SCIENCE & ENGINEERING COOPERATIVE RESEARCH CENTERPURPOSE: ESTABLISH THE CMU/NIST AI MEASUREMENT SCIENCE & ENGINEERING COOPERATIVE RESEARCH CENTER (AIMSEC) TO FACILITATE COLLABORATIVE RESEARCH AND EXPERIMENTATION FOCUSED ON ADVANCING OUR NATIONAL CAPABILITY FOR TEST AND EVALUATION OF MODERN AI CAPABILITIES AND TOOLS. THERE ARE THREE PRINCIPAL GOALS: 1. ADVANCE MODERN AI RISK MANAGEMENT 2. VALIDATE EVALUATION APPROACHES THROUGH STAKEHOLDER PARTNERSHIPS. 3. TRANSLATE ASSESSMENT CAPABILITIES AND METHODOLOGIES TO PRACTICE. RESEARCH WILL FOCUS ON ADVANCING MEASUREMENT SCIENCE FOR MODERN AI SYSTEMS, INCLUDING MACHINE LEARNING (ML) AND GENERATIVE AI (GENAI) SYSTEMS, SUCH AS LARGE LANGUAGE MODELS (LLMS). ACTIVITIES TO BE PERFORMED: CREATE A PRACTICAL TAXONOMY OF THREATS AGAINST LARGE LANGUAGE MODELS (LLMS). THE PROPOSED WORK INCLUDES DEVELOPING TECHNIQUES TO REDUCE MEMORIZATION AND PROTECT CONFIDENTIAL INFORMATION PROVABLY, EVEN IF IT RESULTS IN LESS EXACT OUTPUTS. DEVELOP NEW MEASURES OF ROBUSTNESS FOR ML MODELS THAT BETTER REFLECT A MODEL?S ABILITY TO WITHSTAND ATTACKS IN PRACTICAL SCENARIOS. DEVELOP RICHER DEFINITIONS OF ALIGNMENT, CORRECTNESS, AND INTENT TO DETERMINE THE SUITABILITY OF GENERATIVE AI SYSTEMS FOR THEIR INTENDED USES. CREATE A STATE-OF-THE-ART SUITE OF ATTACKS AGAINST LLMS FOR TESTING NEW MODELS. THESE ACTIVITIES WILL BE ORGANIZED IN THREE STREAMS OF EFFORT ALIGNED WITH NIST AI PRIORITIES. - VALIDITY, RELIABILITY, SAFETY, PRIVACY, AND SECURITY- ACCOUNTABILITY, TRANSPARENCY, FAIRNESS, AND EXPLAINABILITY- GENERATIVE AI EVALUATIONEXPECTED OUTCOMES: DEVELOPMENT OF AI SYSTEM-LEVEL TOOLING, METRICS, EVALUATION PROCEDURES, DEVELOPMENT PROCESSES, AND BEST PRACTICES TO ENSURE AI SYSTEM BUILDERS CONSISTENTLY ENGINEER SAFE AI SYSTEMS. GOALS ARE TO ADVANCE MODERN AI RISK MANAGEMENT, VALIDATE EVALUATION APPROACHES THROUGH STAKEHOLDER PARTNERSHIPS, AND TRANSLATE ASSESSMENT CAPABILITIES AND METHODOLOGIES TO PRACTICE. RESEARCH WILL FOCUS ON ADVANCING MEASUREMENT SCIENCE FOR MODERN AI SYSTEMS, INCLUDING MACHINE LEARNING (ML) AND GENERATIVE AI (GENAI) SYSTEMS, SUCH AS LARGE LANGUAGE MODELS (LLMS). THESE EFFORTS ARE EXPECTED TO YIELD ADDITIONAL APPROACHES TO MITIGATION OF WEAKNESSES AND VULNERABILITIES, PRIVACY PROTECTION, A PRACTICAL TAXONOMY OF THREATS AGAINST LLMS, APPROACHES TO MEET AI GOVERNANCE CHALLENGES, AND NEW SOCIO-TECHNICAL DEFINITIONS AND METRICS.INTENDED BENEFICIARIES: AI STAKEHOLDERS, INCLUDING GOVERNMENT, INDUSTRY, ACADEMIA, AND THE GENERAL PUBLIC, WILL BENEFIT FROM THIS WORK OF ADVANCING CAPABILITY FOR TESTING AND EVALUATING MODERN AI CAPABILITIES AND TOOLS.SUBRECIPIENT ACTIVITIES: THERE ARE NO PLANNED SUBAWARDS., G3562386289, Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), G4010157413, ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), G5367127336, G5464919783, Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI
OpenAlex ID: W7140217467