Evaluating the Feasibility of Local LLMs in Privacy-Sensitive Environments: A Comparative Performance Framework
Year: 2025
Type: dissertation
Abstract: In recent years, large language models (LLMs) have made great advancements in natural language processing (NLP) and other artificial intelligence (AI) applications and have been used in many industries such as healthcare, finance and law, which work with sensitive, confidential and proprietary data. However, most state-of-the-art LLMs are cloud-based, raising concerns about data privacy and security in these sectors. A safe alternative to this could be the implementation of local LLMs, which offer a promising solution by enabling organizations to keep control of their data whilst benefiting from the same AI capabilities. This study evaluates the feasibility of implementing these models in such industries through development of a framework to benchmark LLMs capabilities in diverse NLP tasks. The two most researched local LLMs and the two most researched cloud-based LLMs are tested in this benchmark, to assess the local models’ capabilities, compared to state-of-the-art models. After analyzing the results, we concluded that, despite being competitive in some of the benchmarks, the local LLMs weren’t consistent through the entire evaluation process and consistently underperformed compared to the cloud-based models. These suggest that local models, although not as accurate or versatile as the proprietary LLMs, could be viable for lightweight NLP workloads where latency, cost-efficiency, or privacy are primary concerns. Tasks involving simple fact retrieval or comprehension can often be handled by these models, but they can be relied on for complex and high-stakes applications.
Author: Anjos, Afonso Bruno dos
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7113158353