Considerations for health care institutions training large language models on electronic health records (Preprint)
Year: 2024
Type: preprint
Abstract: BACKGROUND Large language models (LLMs) are usually built (pretrained) on general domain data. To adapt LLMs better to health care needs, it might be beneficial for health care institutions to create LLMs pretrained on their own EHR data. Nevertheless, there are concerns related to the training budget, the type of pretraining (pretraining from scratch vs. pretraining on top of pretrained LLMs), and the EHR database size. OBJECTIVE The objective of this manuscript is to address these concerns and to help decision-makers at health care institutions better understand opportunities and challenges for LLM development. METHODS We use published work on empirical experience in pretraining LLMs, and establish a relationship between the training budget, LLM size, pretraining database size and training time. RESULTS We found that pretraining a modest-sized LLM (13 billion parameters) from scratch requires around 800 GB of EHR data and costs 127,000 USD. For a 65 billion parameter LLM, the EHR requirement is 4000 GB, costing 3 million USD. In comparison, continued pretraining using 500 GB EHR on a 65 billion parameter pretrained LLM would cost 400,000 USD. CONCLUSIONS Building private LLMs on EHR data can be challenging due to budget and database size limitations. Most health care institutions will not have the required resources to train their own LLMs from scratch, and even continued pretraining will be challenging for many. This study provides a framework for health care institutions when deciding about the deployment of LLMs in their systems.
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W4392477433