Year: 2024
Type: dissertation
Abstract: Large Language Models (LLMs) have become increasingly popular for a variety of applications due to their powerful capabilities. While typically run in cloud by third-party providers, there is an increasing interest in running open-source LLMs locally due to privacy concerns and fine-tuning possibilities. The goal of this thesis is find out how feasible code LLMs are to run on limited hardware locally, as well as how well they perform in C++ programming featuring algorithmic problems as well as open-source library usage. Experiments are conducted on a popular coding benchmark HumanEval and a custom C++ dataset created for this thesis with the results evaluated using a \textit{pass@k} metric. Additionally, time is measured for how long these code LLMs need to complete a benchmark to evaluate their average output times on the hardware used. The experiments are run with and without 4-bit quantization to measure if quantization is a feasible method to improve LLM performance on limited hardware. The findings of this thesis show that 4-bit quantization is a useful method to achieve significantly faster output times on LLMs that surpass the hardware's VRAM capacity. In addition, no accuracy loss was noticed in the benchmark results.
Source: Aaltodoc (Aalto University)
Author: Arte, Elias
Institution: Aalto University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7039041544
