Efficiency of small open-source LLMs in software coding : Evaluation of LLMs and consumer hardware
Year: 2024
Type: dissertation
Abstract: Large language models (LLMs) are widely used in many different fields, for example in software development. They can be used to generate and review code which can make development process easier, faster, and cheaper. Problems arise when we need to validate the generated responses. Is the code working and doing what it’s supposed to do? Deployment of LLM requires a lot of processing power and memory from the hardware which needs to be taken into consideration. Can these LLMs be efficiently run with consumer-tier hardware? HumanEval is an open-source dataset that can be used to evaluate LLMs capability in producing efficient and understandable code. It contains both hand-written coding tasks and their respective hand-written solutions. Evaluation is based on comparing the generated response's functionality to the one found in dataset's example solutions. It’s a widely used dataset and many different LLMs include its evaluation scores in their reports and advertisements. To evaluate how well our hardware can run LLMs, we use an open-source benchmark called AI Benchmark Alpha. This benchmark evaluates the hardware’s potential in running different artificial intelligences and their capabilities, including text generation which is the key in code generative AIs. In this work we are evaluating multiple different open-source LLMs that have been advertised as efficient tools in software development, whether through managing project or helping in coding. Through the evaluation scores we can see how they measure up to each other and when compared to big and popular LLMs, like ChatGPT, are they even close to it. Results show that smallsized LLMs, those with three or less billion parameters, should be used with caution. They produce often inaccurate responses which are incapable of solving simple and advanced coding tasks. Important is to notice that provided answer could guide the user towards the right answer even when failing the test case in the benchmark. With current state of consumer hardware, the deployment of LLMs is very limited. To efficiently produce code, users should opt into either making initial investment for hardware that could run larger LLMs or use commercial ones. The latter raises security concerns as the user alone wouldn’t handle their own data anymore.
Author: Jussila, Petrus
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7133024829