Trustworthy NLP for Low-Resource Languages: Agent-Based Uncertainty Modeling for Hebrew Radiology Report Structuring
Year: 2026
Type: conference-paper
Abstract: Reliable extraction of structured information from radiology reports using Large Language Models (LLMs) remains a significant challenge, particularly for complex, non-English texts such as Hebrew.This study proposes an agent-based, uncertainty-aware framework to enhance the reliability and interpretability of LLM predictions in clinical contexts.A total of 9,683 Hebrew radiology reports from Crohn's disease patients (2010-2023) across three medical centers were analyzed.Of these, 512 reports were manually annotated for six gastrointestinal organs and 15 pathological findings, while the remainder were automatically labeled using HSMP-BERT.Structured data extraction was performed with Llama 3.1 (Llama 3-8b-instruct) employing Bayesian Prompt Ensembles (BayesPE), which utilized six semantically equivalent prompts to quantify uncertainty.An Agent-Based Decision Model aggregated prompt outputs into five calibrated confidence levels and was benchmarked against three entropy-based approaches.Model performance was assessed using accuracy, F1 score, precision, recall, and Cohen's Kappa before and after filtering high-uncertainty cases.The agent-based model outperformed all baselines, achieving an F1 score of 0.3967, recall of 0.6437, and Kappa of 0.3006; after excluding cases with uncertainty ≥0.5, the F1 score increased to 0.4787 and Kappa to 0.4258.The proposed framework improves uncertainty calibration and predictive reliability, advancing the safe deployment of LLMs in medical data extraction.
Subfield: Molecular Biology
Domain: Life Sciences
Open Access status: gold
OpenAlex ID: W7166831587
FWCI:
Cites:
Cited by:
Related to: