Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale
Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) can generate or synthesize clinical text for a wide range of applications, from improving clinical documentation to augmenting clinical text analytics.Yet evaluations typically focus on a narrow aspectsuch as similarity or utility comparisons -even though these aspects are complementary and best viewed in parallel.In this study, we aim to conduct a systematic evaluation of LLMgenerated clinical text, which includes intrinsic, extrinsic, and factuality evaluations of synthetic clinical notes rephrased from MIMIC databases at million-note scale.Our analysis demonstrates that synthetic notes preserve core clinical information and predictive utility for coarse-grained tasks despite substantial linguistic changes, but lose fine-grained details for task like ICD coding.We show this loss of detail can be substantially mitigated by rephrasing notes by chunks rather than by the whole note, but at the cost of reduced factual precision under incomplete context.Through fact-checking and error analysis, we further find that synthesis errors are dominated by misinterpretation of clinical context, alongside temporal confusion, measurement errors, and fabricated claims.Finally, we show that the synthetic notes -despite their task-agnostic nature -can effectively augment task-specific training for rare ICD codes.survey of bias in machine learning through the prism of statistical parity.The American statistician, pages 1-11.
Authors: Jinghui Liu, Sarvesh Soni, Anthony Nguyen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7166884358