Year: 2026
Type: conference-paper
Abstract: Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety.These risks are hard to assess: subtle clinical errors are often missed by generic metrics and LLM judges using general criteria, while expert-authored fine-grained rubrics are expensive and difficult to scale.In this paper, we propose a retrievalaugmented multi-agent framework for automatically generating instance-specific evaluation rubrics.Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form fine-grained evaluation criteria.Evaluated on HealthBench and LLMEval-Med, our framework achieves Clinical Intent Alignment (CIA) scores of 50.20% and 31.90%,significantly outperforming the GPT-4o baseline and showing consistent improvements across English and Chinese medical benchmarks.In discriminative tests on HealthBench, our rubrics achieve a 7.8% point higher win rate than GPT-4o and increase the mean score difference from 4.972 to 8.658.Ablation studies further show that individual components contribute differently across datasets, with interaction-intent module providing the most consistent contribution to clinical-criterion coverage.Beyond evaluation, our rubrics guide response refinement, improving response quality by 9.2%.These results suggest that automated, knowledgegrounded rubric generation provides a scalable foundation for evaluating and improving medical LLMs.The code is available at https://anonymous.4open.science/r/ Automated-Rubric-Generation-E716/.
Institution: University College London
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: bronze
OpenAlex ID: W7125424478