Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases
Year: 2026
Type: conference-paper
Abstract: Large language models achieve strong performance on biomedical question answering and summarization benchmarks, yet traditional evaluation metrics often fail to detect clinically significant factual errors.We introduce a unified evaluation framework that combines reference-based measures with evidence-grounded factuality verification to assess biomedical text generation.Evaluating four open-source models across three benchmarks (BioASQ, PubMedQA, MedLFQA), we find that 13.4-24.7% of generated claims are contradicted and 23-41% are unsupported, despite high lexical overlap scores.Our proposed Fact-Aligned Score (FAS) correlates strongly with claim-level verifiability (ρ = 0.68), substantially outperforming ROUGE-L (ρ = 0.41).We release an open-source toolkit with model outputs and analysis scripts to support reproducible factuality evaluation and safer deployment of biomedical LLMs.
Authors: Mahule Roy, Subhas Roy
Institution: University of Oxford
Language: English
Subfield: Molecular Biology
Domain: Life Sciences
Open Access status: gold
OpenAlex ID: W7166822807