Keeping private patient data off the cloud: A comparison of local LLMs for anonymizing radiology reports
Year: 2025
Type: article
Abstract: Purpose To evaluate the performance of open-source, locally deployed large language models (LLMs) in anonymizing radiology reports while preserving clinical data, ensuring compliance with privacy regulations like the Health Insurance Portability and Accountability Act (HIPAA), without relying on cloud-based solutions. Methods Seven state-of-the-art open-source LLMs—Qwen-2.5-coder (7B, 32B), Llama v3.1 (8B, 70B), Llama v3.3 (70B), and Phi3/4 (14B)—were tested on 1000 randomly selected radiology reports (CT, MRI, PET/CT) from a cancer imaging cohort. Models were tasked with generating regular expression (RegEx) rules to remove protected health information (PHI) as defined by HIPAA. Inference was conducted on an A100 GPU and an M4 Max MacBook Pro using the ollama framework. Performance was assessed by three readers for protected health information (PHI) removal accuracy and preservation of non-PHI clinical data, with runtimes recorded. Results Qwen models outperformed others, with Qwen7B achieving a 97.59 % success rate in preserving clinical data (vs. 24.33 % for Llama8B) and Qwen32B achieving 100 % PHI removal (vs. 1.91 % patient names missed by Llama v3.1 70B). Llama models frequently misidentified clinical numbers as PHI, while Phi models produced unusable results due to overly aggressive rules or hallucinations. Runtimes were significantly shorter for Qwen models (e.g., Qwen7B: 2h50′ on A100 vs. Llama8B: 3h30′). Conclusion Locally deployed Qwen-2.5-coder models offer a superior, privacy-preserving solution for anonymizing radiology reports, balancing PHI removal with clinical data retention. This approach supports scalable, compliant anonymization of radiology reports for research without cloud dependency, outperforming other current state-of-the-art local LLMs.
Institutions: New York University, NYU Langone Health, Health First
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $2,900
OpenAlex ID: W4409839694