Prompt Distillation for Clinical Guideline Knowledge Injection: Evaluating Multi-Agent Deployment in Clinical Decision Support
Year: 2026
Type: dissertation
Abstract: Large language models offer strong potential for clinical decision support, but injecting up-to-date clinical guideline knowledge into them without expensive retraining remains an open problem. This thesis investigates whether prompt distillation, a parameter efficient knowledge injection method, can internalize clinical guideline knowledge in a medical large language model and, more importantly, whether that knowledge translates into better decisions when the model is deployed as the reasoning agent in a multi-agent clinical decision support system. Two variants of an 8B-parameter medical model were prompt-distilled on approximately 37,000 EPFL clinical guideline documents: PD-Only, and PD+Tülu, regularized with Tülu 3 instruction data. Stage 1 evaluated them on held-out guideline questions under closed-book and retrieval-augmented settings; Stage 2 embedded them as the Decision Maker agent in a four-agent pipeline and evaluated the resulting systems on 500 MIMIC-IV admissions using a composite Clinical Quality Score (CQS, max 50). The findings are threefold. First, prompt distillation injected guideline knowledge effectively and scaled non-linearly with training questions per document. Under retrieval, PD+Tülu integrated context best, achieving the highest held-out guideline-question score (3.98 versus 3.78 for the base model). Downstream, however, neither variant beat the base model: the base system scored highest (CQS 35.77), PD+Tülu reached 34.78, and PD-Only degraded sharply to 31.09, with its reasoning-failure rate rising to 34% against the base model’s 22%. The decisive factor was not how much knowledge a model absorbed but how much general capability it retained while doing so. These results show that intrinsic question-answering metrics are insufficient for evaluating clinical LLMs intended for multi-agent deployment, and that the trade-off between domain specialization and general capability warrants explicit treatment in future work.
Source: Aaltodoc (Aalto University)
Author: Phan, Anh
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7204019538